2024-03-26T21:42:33.930 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T21:42:33.931 DEBUG:teuthology.run:Teuthology command: teuthology --name gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176 --verbose -- /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/orig.config.yaml 2024-03-26T21:42:33.983 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T21:42:34.083 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176 branch: wip-guits-testing-2024-03-26-0925 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: gabrioux@ibm.com first_in_suite: false job_id: '7624176' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-03-26-0925 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-guits-testing-2024-03-26-0925 sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 832 sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-03-26-0925 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 targets: smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuJlGUFZj0dqznIhEZ+MceqG2b/P6HvERcGC9PK+u7wG0xCR7x5k53UR19rNTkBH3/aNqe5B4Eu8pSjuNQzyv4= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH1ye03rR2irGWQ5NmnAkqEhkvtxrNZ/v8LYn1Nrd71KYevo/lz2SNIUvLg5SRSG/Q3dLxVTM5IhWiYgIUK/ilQ= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_20:31:35 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-26T21:42:34.083 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa; will attempt to use it 2024-03-26T21:42:34.083 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks 2024-03-26T21:42:34.084 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T21:42:34.085 INFO:teuthology.task.internal:Checking packages... 2024-03-26T21:42:34.108 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '42fc52b86b14d4f88c40d75eb7102e0ab3918f55' 2024-03-26T21:42:34.108 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:42:34.108 INFO:teuthology.packaging:ref: None 2024-03-26T21:42:34.108 INFO:teuthology.packaging:tag: None 2024-03-26T21:42:34.108 INFO:teuthology.packaging:branch: wip-guits-testing-2024-03-26-0925 2024-03-26T21:42:34.108 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:42:34.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-03-26-0925 2024-03-26T21:42:34.250 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2484.g42fc52b8 2024-03-26T21:42:34.251 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T21:42:34.257 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T21:42:34.257 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T21:42:34.263 INFO:teuthology.task.internal:Saving configuration 2024-03-26T21:42:34.283 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T21:42:34.289 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T21:42:34.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:36:13.192283', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuJlGUFZj0dqznIhEZ+MceqG2b/P6HvERcGC9PK+u7wG0xCR7x5k53UR19rNTkBH3/aNqe5B4Eu8pSjuNQzyv4='} 2024-03-26T21:42:34.331 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:36:13.193501', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH1ye03rR2irGWQ5NmnAkqEhkvtxrNZ/v8LYn1Nrd71KYevo/lz2SNIUvLg5SRSG/Q3dLxVTM5IhWiYgIUK/ilQ='} 2024-03-26T21:42:34.332 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T21:42:34.339 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-26T21:42:34.339 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-26T21:42:34.339 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T21:42:34.412 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7d5a8d0d0>, signals=[15]) 2024-03-26T21:42:34.412 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T21:42:34.418 INFO:teuthology.task.internal:Opening connections... 2024-03-26T21:42:34.418 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-03-26T21:42:34.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:42:34.523 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-26T21:42:34.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:42:34.625 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T21:42:34.632 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-03-26T21:42:34.650 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-03-26T21:42:34.651 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-03-26T21:42:34.707 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-03-26T21:42:34.707 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T21:42:34.708 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T21:42:34.709 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-03-26T21:42:34.735 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-26T21:42:34.752 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-26T21:42:34.752 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-26T21:42:34.809 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-26T21:42:34.809 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-26T21:42:34.809 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-26T21:42:34.809 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T21:42:34.810 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T21:42:34.810 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-26T21:42:34.843 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T21:42:34.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T21:42:34.858 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T21:42:34.858 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-03-26T21:42:34.860 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-26T21:42:34.876 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T21:42:34.882 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T21:42:34.882 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-03-26T21:42:34.917 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-26T21:42:35.041 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T21:42:35.094 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T21:42:35.109 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T21:42:35.109 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T21:42:35.109 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T21:42:35.110 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T21:42:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.126 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-26T21:42:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.127 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-26T21:42:35.184 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-03-26T21:42:35.184 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-03-26T21:42:35.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-03-26T21:42:35.185 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-361.el9.x86_64 2024-03-26T21:42:35.185 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-361.el9.x86_64 2024-03-26T21:42:35.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-03-26T21:42:37.347 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T21:42:37.371 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-26T21:42:38.747 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-26T21:42:39.094 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-26T21:42:39.962 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T21:42:40.713 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-03-26T21:42:42.531 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-26T21:42:47.938 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:06 2024-03-26T21:42:48.514 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-26T21:42:50.781 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-26T21:42:51.955 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-26T21:42:52.235 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T21:42:53.548 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-26T21:42:55.106 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-26T21:42:57.274 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-26T21:42:59.430 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T21:42:59.430 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T21:42:59.573 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:42:59.583 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-03-26T21:42:59.584 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:42:59.585 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 76 M 2024-03-26T21:42:59.585 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-03-26T21:42:59.585 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:42:59.800 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 636 kB/s | 66 kB 00:00 2024-03-26T21:43:00.148 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 9.2 kB/s | 1.7 kB 00:00 2024-03-26T21:43:00.683 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-03-26T21:43:01.592 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 15 MB/s | 27 MB 00:01 2024-03-26T21:43:02.042 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 14 MB/s | 33 MB 00:02 2024-03-26T21:43:02.042 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:43:02.042 INFO:teuthology.orchestra.run.smithi196.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-26T21:43:02.288 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T21:43:02.289 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T21:43:02.389 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:43:02.393 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:43:02.393 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-03-26T21:43:02.394 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:02.395 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 76 M 2024-03-26T21:43:02.395 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 123 M 2024-03-26T21:43:02.395 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:43:02.616 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 404 kB/s | 66 kB 00:00 2024-03-26T21:43:02.710 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:43:03.118 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:43:03.118 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:43:04.140 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:43:04.141 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:43:05.032 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 5.8 MB/s | 15 MB 00:02 2024-03-26T21:43:06.601 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:43:07.206 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:43:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:43:08.315 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 4.8 MB/s | 27 MB 00:05 2024-03-26T21:43:09.982 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 4.4 MB/s | 33 MB 00:07 2024-03-26T21:43:09.982 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:43:09.982 INFO:teuthology.orchestra.run.smithi194.stdout:Total 10 MB/s | 76 MB 00:07 2024-03-26T21:43:10.036 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:43:10.177 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:43:10.651 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:43:11.013 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:43:11.014 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:43:12.032 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:43:12.032 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:43:14.513 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:43:14.860 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:43:14.969 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:43:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:43:17.932 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:43:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:43:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:22.804 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:43:23.042 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:27.788 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:49.036 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:49.041 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T21:43:49.042 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:43:49.080 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:49.437 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:49.437 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:43:49.437 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:43:49.437 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:43:49.850 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:43:50.165 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:43:50.262 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:43:50.262 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:43:50.262 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T21:43:50.263 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T21:43:50.263 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T21:43:50.263 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-26T21:43:50.263 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T21:43:50.263 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:43:50.342 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:43:50.342 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:43:50.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-03-26T21:43:50.381 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T21:43:50.382 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:43:50.383 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T21:43:50.384 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T21:43:50.384 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-432.el9.x86_64 2024-03-26T21:43:50.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T21:43:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-03-26T21:43:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T21:43:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-03-26T21:43:52.476 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-26T21:43:52.491 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.EYdF0rE9sE 2024-03-26T21:43:52.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EYdF0rE9sE 2024-03-26T21:43:52.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.EYdF0rE9sE 2024-03-26T21:43:52.699 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.EYdF0rE9sE is 6KB 2024-03-26T21:43:52.718 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.EYdF0rE9sE 2024-03-26T21:43:52.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T21:43:52.797 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T21:43:52.797 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T21:43:52.797 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T21:43:52.797 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T21:43:52.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T21:43:52.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-03-26T21:43:56.961 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:56.962 INFO:teuthology.orchestra.run.smithi194.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T21:43:56.962 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:57.012 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:57.389 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:57.389 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:43:57.389 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:43:57.390 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:43:57.798 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:43:57.798 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:57.799 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:43:58.116 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:43:58.234 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:43:58.235 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:43:58.235 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T21:43:58.235 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T21:43:58.235 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T21:43:58.235 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-26T21:43:58.235 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T21:43:58.235 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:43:58.314 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:43:58.315 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:43:58.315 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-03-26T21:43:58.352 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-03-26T21:43:58.352 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-03-26T21:43:58.352 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-03-26T21:43:58.352 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 68.el9 2024-03-26T21:43:58.352 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8264254 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-03-26T21:43:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:43:58.354 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T21:43:58.355 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T21:43:58.355 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-432.el9.x86_64 2024-03-26T21:43:58.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T21:43:59.003 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-03-26T21:44:00.419 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T21:44:00.440 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-03-26T21:44:00.442 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-03-26T21:44:00.497 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.RvYFBBnF77 2024-03-26T21:44:00.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RvYFBBnF77 2024-03-26T21:44:00.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.RvYFBBnF77 2024-03-26T21:44:00.711 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.RvYFBBnF77 is 6KB 2024-03-26T21:44:00.758 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.RvYFBBnF77 2024-03-26T21:44:00.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T21:44:00.837 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T21:44:00.837 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T21:44:00.837 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T21:44:00.837 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T21:44:00.838 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T21:44:00.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-03-26T21:44:22.981 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-26T21:44:22.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-26T21:44:22.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:44:31.002 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-26T21:44:31.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-26T21:44:31.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:44:41.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-03-26T21:44:49.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-03-26T21:44:50.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-26T21:44:50.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:44:58.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-26T21:44:58.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:45:01.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-03-26T21:45:13.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-26T21:45:13.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:45:45.422 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-26T21:45:45.813 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-26T21:45:45.813 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T21:45:45.814 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-26T21:45:45.828 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-432.el9.x86_64 2024-03-26T21:45:45.828 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T21:45:45.828 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T21:45:45.828 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T21:45:46.830 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-26T21:45:46.830 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T21:45:46.909 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-03-26T21:45:46.968 DEBUG:teuthology.parallel:result is None 2024-03-26T21:45:50.508 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T21:46:02.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-26T21:46:02.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:46:02.992 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-26T21:46:03.364 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-26T21:46:03.364 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T21:46:03.365 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-26T21:46:03.379 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-432.el9.x86_64 2024-03-26T21:46:03.379 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T21:46:03.380 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T21:46:03.380 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T21:46:04.380 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-26T21:46:04.380 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T21:46:04.469 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-03-26T21:46:04.506 DEBUG:teuthology.parallel:result is None 2024-03-26T21:46:04.506 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T21:46:04.513 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T21:46:04.513 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T21:46:04.516 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T21:46:04.535 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T21:46:04.541 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T21:46:04.550 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T21:46:04.550 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T21:46:04.575 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T21:46:04.669 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T21:46:04.675 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T21:46:04.676 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T21:46:04.678 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T21:46:04.709 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:46:04.716 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:46:04.723 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:46:04.730 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:46:04.732 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T21:46:04.739 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T21:46:04.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T21:46:04.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T21:46:04.803 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T21:46:04.813 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T21:46:04.813 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T21:46:04.837 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T21:46:04.860 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:46:04.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:46:04.987 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:46:05.070 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:46:05.112 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:46:05.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T21:46:05.179 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:46:05.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:46:05.276 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:46:05.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:46:05.407 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:46:05.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T21:46:05.476 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-03-26T21:46:05.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-26T21:46:05.525 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T21:46:05.574 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T21:46:06.038 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T21:46:06.046 INFO:teuthology.task.internal:Starting timer... 2024-03-26T21:46:06.047 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T21:46:06.058 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T21:46:06.066 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-26T21:46:06.066 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-03-26T21:46:06.121 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-03-26T21:46:06.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-26T21:46:06.176 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-26T21:46:06.179 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T21:46:06.179 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-03-26T21:46:06.206 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-03-26T21:46:06.207 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-26T21:46:06.235 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-26T21:46:06.235 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi194.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-26T21:46:06.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T21:46:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T21:46:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.304 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T21:46:06.304 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-03-26T21:46:06.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-26T21:46:06.395 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T21:46:06.405 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T21:46:06.423 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T21:46:06.424 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi194.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T21:50:23.956 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi194.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-26T21:50:23.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-26T21:50:23.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:50:24.051 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-26T21:50:24.133 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-26T21:50:24.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-26T21:50:24.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:50:24.222 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-26T21:50:24.298 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-26T21:50:24.298 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T21:50:24.308 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T21:50:24.308 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T21:50:24.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T21:50:24.311 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T21:50:24.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T21:50:24.340 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T21:50:24.357 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T21:50:24.369 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T21:50:24.385 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T21:50:24.386 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-03-26T21:50:24.399 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-03-26T21:50:24.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-26T21:50:24.416 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T21:50:24.427 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-26T21:50:24.432 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T21:50:24.444 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T21:50:24.460 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T21:50:24.488 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-26T21:50:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T21:50:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-26T21:50:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.513 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-26T21:50:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T21:50:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-26T21:50:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.517 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:50:24.517 INFO:teuthology.run_tasks:Running task install... 2024-03-26T21:50:24.525 DEBUG:teuthology.task.install:project ceph 2024-03-26T21:50:24.525 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'}} 2024-03-26T21:50:24.525 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'} 2024-03-26T21:50:24.526 INFO:teuthology.task.install:Using flavor: default 2024-03-26T21:50:24.531 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T21:50:24.531 INFO:teuthology.task.install:extra packages: [] 2024-03-26T21:50:24.532 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'tag': None, 'wait_for_package': False} 2024-03-26T21:50:24.532 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:50:24.532 INFO:teuthology.packaging:ref: None 2024-03-26T21:50:24.532 INFO:teuthology.packaging:tag: None 2024-03-26T21:50:24.532 INFO:teuthology.packaging:branch: reef 2024-03-26T21:50:24.532 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:50:24.533 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-26T21:50:24.535 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'tag': None, 'wait_for_package': False} 2024-03-26T21:50:24.535 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:50:24.535 INFO:teuthology.packaging:ref: None 2024-03-26T21:50:24.535 INFO:teuthology.packaging:tag: None 2024-03-26T21:50:24.535 INFO:teuthology.packaging:branch: reef 2024-03-26T21:50:24.535 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:50:24.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-26T21:50:24.790 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/ 2024-03-26T21:50:24.790 INFO:teuthology.task.install.rpm:Package version is 18.2.2-983.gd83b58fa 2024-03-26T21:50:24.805 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/ 2024-03-26T21:50:24.805 INFO:teuthology.task.install.rpm:Package version is 18.2.2-983.gd83b58fa 2024-03-26T21:50:24.913 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T21:50:24.914 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:50:24.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T21:50:24.942 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T21:50:24.942 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:50:24.942 INFO:teuthology.packaging:ref: None 2024-03-26T21:50:24.942 INFO:teuthology.packaging:tag: None 2024-03-26T21:50:24.942 INFO:teuthology.packaging:branch: reef 2024-03-26T21:50:24.942 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:50:24.942 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T21:50:24.977 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T21:50:24.978 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:50:24.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T21:50:25.005 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T21:50:25.005 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:50:25.005 INFO:teuthology.packaging:ref: None 2024-03-26T21:50:25.006 INFO:teuthology.packaging:tag: None 2024-03-26T21:50:25.006 INFO:teuthology.packaging:branch: reef 2024-03-26T21:50:25.006 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:50:25.006 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T21:50:25.009 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T21:50:25.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T21:50:25.089 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T21:50:25.152 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T21:50:25.156 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-03-26T21:50:25.158 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-03-26T21:50:25.218 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-26T21:50:25.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-26T21:50:25.405 INFO:teuthology.orchestra.run.smithi194.stdout:74 files removed 2024-03-26T21:50:25.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-03-26T21:50:25.463 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-26T21:50:25.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-03-26T21:50:26.143 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 192 kB/s | 77 kB 00:00 2024-03-26T21:50:26.201 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 187 kB/s | 77 kB 00:00 2024-03-26T21:50:26.536 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 29 kB/s | 9.9 kB 00:00 2024-03-26T21:50:26.627 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 26 kB/s | 9.9 kB 00:00 2024-03-26T21:50:26.936 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-26T21:50:27.002 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-26T21:50:27.430 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-26T21:50:27.672 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-26T21:50:28.855 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-26T21:50:29.380 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-26T21:50:30.212 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T21:50:30.779 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-26T21:50:32.105 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-26T21:50:36.481 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.1 MB 00:00 2024-03-26T21:50:38.057 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-26T21:50:40.244 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-26T21:50:45.125 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-26T21:50:47.602 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 7.7 M 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-03-26T21:50:47.605 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 3.3 M 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 3.0 M 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 5.1 M 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 18 M 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 25 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 702 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T21:50:47.606 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 474 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 4.5 M 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 45 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 129 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 162 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 321 k 2024-03-26T21:50:47.607 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 298 k 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 99 k 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout:Install 28 Packages 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:50:47.608 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 53 M 2024-03-26T21:50:47.609 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:50:48.450 INFO:teuthology.orchestra.run.smithi194.stdout:(1/30): ceph-radosgw-18.2.2-983.gd83b58fa.el9.x 10 MB/s | 7.7 MB 00:00 2024-03-26T21:50:48.526 INFO:teuthology.orchestra.run.smithi194.stdout:(2/30): ceph-base-18.2.2-983.gd83b58fa.el9.x86_ 6.1 MB/s | 5.1 MB 00:00 2024-03-26T21:50:48.551 INFO:teuthology.orchestra.run.smithi194.stdout:(3/30): ceph-selinux-18.2.2-983.gd83b58fa.el9.x 251 kB/s | 25 kB 00:00 2024-03-26T21:50:48.601 INFO:teuthology.orchestra.run.smithi194.stdout:(4/30): libcephfs2-18.2.2-983.gd83b58fa.el9.x86 9.1 MB/s | 702 kB 00:00 2024-03-26T21:50:48.626 INFO:teuthology.orchestra.run.smithi194.stdout:(5/30): libradosstriper1-18.2.2-983.gd83b58fa.e 6.2 MB/s | 474 kB 00:00 2024-03-26T21:50:48.677 INFO:teuthology.orchestra.run.smithi194.stdout:(6/30): python3-ceph-argparse-18.2.2-983.gd83b5 900 kB/s | 45 kB 00:00 2024-03-26T21:50:48.735 INFO:teuthology.orchestra.run.smithi194.stdout:(7/30): python3-ceph-common-18.2.2-983.gd83b58f 2.2 MB/s | 129 kB 00:00 2024-03-26T21:50:48.794 INFO:teuthology.orchestra.run.smithi194.stdout:(8/30): python3-cephfs-18.2.2-983.gd83b58fa.el9 2.7 MB/s | 162 kB 00:00 2024-03-26T21:50:48.860 INFO:teuthology.orchestra.run.smithi194.stdout:(9/30): python3-rados-18.2.2-983.gd83b58fa.el9. 4.7 MB/s | 321 kB 00:00 2024-03-26T21:50:48.919 INFO:teuthology.orchestra.run.smithi194.stdout:(10/30): librgw2-18.2.2-983.gd83b58fa.el9.x86_6 14 MB/s | 4.5 MB 00:00 2024-03-26T21:50:48.945 INFO:teuthology.orchestra.run.smithi194.stdout:(11/30): python3-rbd-18.2.2-983.gd83b58fa.el9.x 3.5 MB/s | 298 kB 00:00 2024-03-26T21:50:48.970 INFO:teuthology.orchestra.run.smithi194.stdout:(12/30): python3-rgw-18.2.2-983.gd83b58fa.el9.x 1.9 MB/s | 99 kB 00:00 2024-03-26T21:50:49.329 INFO:teuthology.orchestra.run.smithi194.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 276 kB/s | 106 kB 00:00 2024-03-26T21:50:49.404 INFO:teuthology.orchestra.run.smithi194.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 367 kB/s | 159 kB 00:00 2024-03-26T21:50:49.638 INFO:teuthology.orchestra.run.smithi194.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 661 kB 00:00 2024-03-26T21:50:49.662 INFO:teuthology.orchestra.run.smithi194.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 44 kB 00:00 2024-03-26T21:50:49.796 INFO:teuthology.orchestra.run.smithi194.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-26T21:50:49.822 INFO:teuthology.orchestra.run.smithi194.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 259 kB/s | 41 kB 00:00 2024-03-26T21:50:50.197 INFO:teuthology.orchestra.run.smithi194.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 277 kB/s | 111 kB 00:00 2024-03-26T21:50:50.297 INFO:teuthology.orchestra.run.smithi194.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 650 kB/s | 309 kB 00:00 2024-03-26T21:50:50.397 INFO:teuthology.orchestra.run.smithi194.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-26T21:50:50.498 INFO:teuthology.orchestra.run.smithi194.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-26T21:50:50.598 INFO:teuthology.orchestra.run.smithi194.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-26T21:50:50.723 INFO:teuthology.orchestra.run.smithi194.stdout:(24/30): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-03-26T21:50:50.782 INFO:teuthology.orchestra.run.smithi194.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-26T21:50:50.824 INFO:teuthology.orchestra.run.smithi194.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-26T21:50:50.958 INFO:teuthology.orchestra.run.smithi194.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-26T21:50:51.141 INFO:teuthology.orchestra.run.smithi194.stdout:(28/30): librados2-18.2.2-983.gd83b58fa.el9.x86 10 MB/s | 3.3 MB 00:00 2024-03-26T21:50:51.299 INFO:teuthology.orchestra.run.smithi194.stdout:(29/30): librbd1-18.2.2-983.gd83b58fa.el9.x86_6 8.8 MB/s | 3.0 MB 00:00 2024-03-26T21:50:52.393 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 941 kB/s | 19 MB 00:20 2024-03-26T21:50:53.233 INFO:teuthology.orchestra.run.smithi194.stdout:(30/30): ceph-common-18.2.2-983.gd83b58fa.el9.x 3.3 MB/s | 18 MB 00:05 2024-03-26T21:50:53.237 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:50:53.237 INFO:teuthology.orchestra.run.smithi194.stdout:Total 9.4 MB/s | 53 MB 00:05 2024-03-26T21:50:53.238 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:50:53.273 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:50:53.274 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:50:53.727 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:50:53.727 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:50:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:50:55.166 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-26T21:50:55.204 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-26T21:50:55.254 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-26T21:50:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-26T21:50:55.636 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 5/32 2024-03-26T21:50:55.728 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 5/32 2024-03-26T21:50:55.813 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 6/32 2024-03-26T21:50:55.846 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 7/32 2024-03-26T21:50:55.900 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 7/32 2024-03-26T21:50:55.930 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-26T21:50:55.981 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:18.2.2-983.gd83b58fa.el9.x 9/32 2024-03-26T21:50:56.048 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 10/32 2024-03-26T21:50:56.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 11/32 2024-03-26T21:50:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 11/32 2024-03-26T21:50:56.252 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-26T21:50:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-26T21:50:56.322 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-26T21:50:56.346 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-26T21:50:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-26T21:50:56.414 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-26T21:50:56.453 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-26T21:50:56.774 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-26T21:50:56.804 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 20/32 2024-03-26T21:50:56.878 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 20/32 2024-03-26T21:50:56.909 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 21/32 2024-03-26T21:50:56.962 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-26T21:50:57.379 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-26T21:50:57.407 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 24/32 2024-03-26T21:50:57.494 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 24/32 2024-03-26T21:50:57.562 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 25/32 2024-03-26T21:50:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:18.2.2-983.gd83b58fa.el9.x86 26/32 2024-03-26T21:50:57.661 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-26T21:50:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 27/32 2024-03-26T21:50:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 27/32 2024-03-26T21:51:00.169 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-26T21:51:00.224 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 27/32 2024-03-26T21:51:00.259 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 28/32 2024-03-26T21:51:00.441 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 28/32 2024-03-26T21:51:00.441 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T21:51:00.441 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T21:51:00.442 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:00.488 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 29/32 2024-03-26T21:51:01.640 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T21:51:06.571 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-26T21:51:09.065 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:51:09.074 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 7.7 M 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 3.3 M 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 3.0 M 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 5.1 M 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 18 M 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 25 k 2024-03-26T21:51:09.075 INFO:teuthology.orchestra.run.smithi196.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 702 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 474 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T21:51:09.076 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 4.5 M 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 45 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 129 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 162 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 321 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 298 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 99 k 2024-03-26T21:51:09.077 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout:Install 28 Packages 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-03-26T21:51:09.078 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:09.079 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 53 M 2024-03-26T21:51:09.079 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:51:09.955 INFO:teuthology.orchestra.run.smithi196.stdout:(1/30): ceph-base-18.2.2-983.gd83b58fa.el9.x86_ 6.1 MB/s | 5.1 MB 00:00 2024-03-26T21:51:10.005 INFO:teuthology.orchestra.run.smithi196.stdout:(2/30): ceph-selinux-18.2.2-983.gd83b58fa.el9.x 505 kB/s | 25 kB 00:00 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 29/32 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-03-26T21:51:10.080 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:10.105 INFO:teuthology.orchestra.run.smithi196.stdout:(3/30): libcephfs2-18.2.2-983.gd83b58fa.el9.x86 6.9 MB/s | 702 kB 00:00 2024-03-26T21:51:10.214 INFO:teuthology.orchestra.run.smithi196.stdout:(4/30): libradosstriper1-18.2.2-983.gd83b58fa.e 4.3 MB/s | 474 kB 00:00 2024-03-26T21:51:10.798 INFO:teuthology.orchestra.run.smithi196.stdout:(5/30): librgw2-18.2.2-983.gd83b58fa.el9.x86_64 7.6 MB/s | 4.5 MB 00:00 2024-03-26T21:51:10.842 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 30/32 2024-03-26T21:51:10.867 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 30/32 2024-03-26T21:51:10.867 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:10.867 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T21:51:10.867 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:51:10.868 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:51:10.868 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:10.869 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T21:51:10.953 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T21:51:10.953 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T21:51:10.973 INFO:teuthology.orchestra.run.smithi196.stdout:(6/30): ceph-common-18.2.2-983.gd83b58fa.el9.x8 9.9 MB/s | 18 MB 00:01 2024-03-26T21:51:10.998 INFO:teuthology.orchestra.run.smithi196.stdout:(7/30): python3-ceph-argparse-18.2.2-983.gd83b5 224 kB/s | 45 kB 00:00 2024-03-26T21:51:11.024 INFO:teuthology.orchestra.run.smithi196.stdout:(8/30): python3-ceph-common-18.2.2-983.gd83b58f 2.5 MB/s | 129 kB 00:00 2024-03-26T21:51:11.049 INFO:teuthology.orchestra.run.smithi196.stdout:(9/30): python3-cephfs-18.2.2-983.gd83b58fa.el9 3.1 MB/s | 162 kB 00:00 2024-03-26T21:51:11.074 INFO:teuthology.orchestra.run.smithi196.stdout:(10/30): python3-rados-18.2.2-983.gd83b58fa.el9 6.2 MB/s | 321 kB 00:00 2024-03-26T21:51:11.166 INFO:teuthology.orchestra.run.smithi196.stdout:(11/30): ceph-radosgw-18.2.2-983.gd83b58fa.el9. 3.8 MB/s | 7.7 MB 00:02 2024-03-26T21:51:11.191 INFO:teuthology.orchestra.run.smithi196.stdout:(12/30): python3-rbd-18.2.2-983.gd83b58fa.el9.x 2.0 MB/s | 298 kB 00:00 2024-03-26T21:51:11.217 INFO:teuthology.orchestra.run.smithi196.stdout:(13/30): python3-rgw-18.2.2-983.gd83b58fa.el9.x 698 kB/s | 99 kB 00:00 2024-03-26T21:51:11.367 INFO:teuthology.orchestra.run.smithi196.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 294 kB/s | 44 kB 00:00 2024-03-26T21:51:11.392 INFO:teuthology.orchestra.run.smithi196.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2024-03-26T21:51:11.434 INFO:teuthology.orchestra.run.smithi196.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 657 kB/s | 159 kB 00:00 2024-03-26T21:51:11.468 INFO:teuthology.orchestra.run.smithi196.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-03-26T21:51:11.493 INFO:teuthology.orchestra.run.smithi196.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-26T21:51:11.518 INFO:teuthology.orchestra.run.smithi196.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 496 kB/s | 41 kB 00:00 2024-03-26T21:51:11.735 INFO:teuthology.orchestra.run.smithi196.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 415 kB/s | 111 kB 00:00 2024-03-26T21:51:11.802 INFO:teuthology.orchestra.run.smithi196.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-26T21:51:11.827 INFO:teuthology.orchestra.run.smithi196.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-26T21:51:11.869 INFO:teuthology.orchestra.run.smithi196.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 725 kB/s | 49 kB 00:00 2024-03-26T21:51:11.886 INFO:teuthology.orchestra.run.smithi196.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-26T21:51:12.003 INFO:teuthology.orchestra.run.smithi196.stdout:(25/30): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-26T21:51:12.028 INFO:teuthology.orchestra.run.smithi196.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 1/32 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 2/32 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 3/32 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 4/32 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 5/32 2024-03-26T21:51:12.050 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 6/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 7/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:18.2.2-983.gd83b58fa.el9.x 8/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:18.2.2-983.gd83b58fa.el9.x86 9/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 10/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 11/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 12/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 13/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-26T21:51:12.051 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-26T21:51:12.052 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-26T21:51:12.053 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-26T21:51:12.053 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 29/32 2024-03-26T21:51:12.053 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-26T21:51:12.053 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 31/32 2024-03-26T21:51:12.087 INFO:teuthology.orchestra.run.smithi196.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-26T21:51:12.112 INFO:teuthology.orchestra.run.smithi196.stdout:(28/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-26T21:51:12.362 INFO:teuthology.orchestra.run.smithi196.stdout:(29/30): librbd1-18.2.2-983.gd83b58fa.el9.x86_6 11 MB/s | 3.0 MB 00:00 2024-03-26T21:51:12.421 INFO:teuthology.orchestra.run.smithi196.stdout:(30/30): librados2-18.2.2-983.gd83b58fa.el9.x86 8.4 MB/s | 3.3 MB 00:00 2024-03-26T21:51:12.425 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:12.425 INFO:teuthology.orchestra.run.smithi196.stdout:Total 16 MB/s | 53 MB 00:03 2024-03-26T21:51:12.426 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:51:12.460 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:51:12.460 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:51:12.925 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:51:12.925 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:51:13.077 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T21:51:13.077 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T21:51:13.078 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:13.080 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T21:51:13.080 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T21:51:13.080 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:13.080 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:13.424 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-03-26T21:51:13.987 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:14.138 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:14.138 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 40 M 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 166 k 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:14.139 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:14.140 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-03-26T21:51:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:14.140 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 41 M 2024-03-26T21:51:14.140 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 165 M 2024-03-26T21:51:14.140 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:51:14.378 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-26T21:51:14.383 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-26T21:51:14.408 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-26T21:51:14.441 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-26T21:51:14.459 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-26T21:51:14.475 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): libcephsqlite-18.2.2-983.gd83b58fa.el9.x 595 kB/s | 166 kB 00:00 2024-03-26T21:51:14.500 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-26T21:51:14.642 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-26T21:51:14.824 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-26T21:51:14.858 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 5/32 2024-03-26T21:51:14.951 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 5/32 2024-03-26T21:51:15.034 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 6/32 2024-03-26T21:51:15.076 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 7/32 2024-03-26T21:51:15.138 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 7/32 2024-03-26T21:51:15.167 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-26T21:51:15.205 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:18.2.2-983.gd83b58fa.el9.x 9/32 2024-03-26T21:51:15.276 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 10/32 2024-03-26T21:51:15.303 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 11/32 2024-03-26T21:51:15.369 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 11/32 2024-03-26T21:51:15.476 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-26T21:51:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-26T21:51:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-26T21:51:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-26T21:51:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-26T21:51:15.646 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-26T21:51:15.671 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-26T21:51:15.970 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-26T21:51:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 20/32 2024-03-26T21:51:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 20/32 2024-03-26T21:51:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 21/32 2024-03-26T21:51:16.190 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-26T21:51:16.201 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): ceph-test-18.2.2-983.gd83b58fa.el9.x86_6 20 MB/s | 40 MB 00:02 2024-03-26T21:51:16.203 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:16.204 INFO:teuthology.orchestra.run.smithi194.stdout:Total 20 MB/s | 41 MB 00:02 2024-03-26T21:51:16.218 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:16.239 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:16.239 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:16.570 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:16.570 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:16.593 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-26T21:51:16.620 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 24/32 2024-03-26T21:51:16.707 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 24/32 2024-03-26T21:51:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 25/32 2024-03-26T21:51:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:18.2.2-983.gd83b58fa.el9.x86 26/32 2024-03-26T21:51:17.089 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:17.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T21:51:17.260 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T21:51:17.296 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:51:17.335 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T21:51:17.362 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 5/6 2024-03-26T21:51:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 27/32 2024-03-26T21:51:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 27/32 2024-03-26T21:51:19.496 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 27/32 2024-03-26T21:51:19.538 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 28/32 2024-03-26T21:51:19.643 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 28/32 2024-03-26T21:51:19.643 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T21:51:19.643 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T21:51:19.643 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 29/32 2024-03-26T21:51:21.077 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 5/6 2024-03-26T21:51:21.154 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 6/6 2024-03-26T21:51:23.173 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 6/6 2024-03-26T21:51:23.174 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 1/6 2024-03-26T21:51:23.174 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 2/6 2024-03-26T21:51:23.174 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:51:23.174 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T21:51:23.174 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T21:51:23.600 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T21:51:23.600 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:23.600 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:23.600 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout: jq-1.6-16.el9.x86_64 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:23.601 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:23.869 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-03-26T21:51:24.442 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:39 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:24.598 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 6.5 k 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:51:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 2.1 M 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 1.4 M 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 245 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 4.7 M 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 16 M 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T21:51:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T21:51:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout:Install 33 Packages 2024-03-26T21:51:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:24.604 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 29 M 2024-03-26T21:51:24.604 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 104 M 2024-03-26T21:51:24.604 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:25.179 INFO:teuthology.orchestra.run.smithi194.stdout:(1/33): ceph-18.2.2-983.gd83b58fa.el9.x86_64.rp 39 kB/s | 6.5 kB 00:00 2024-03-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi194.stdout:(2/33): ceph-mgr-18.2.2-983.gd83b58fa.el9.x86_6 4.3 MB/s | 1.4 MB 00:00 2024-03-26T21:51:25.763 INFO:teuthology.orchestra.run.smithi194.stdout:(3/33): ceph-mon-18.2.2-983.gd83b58fa.el9.x86_6 8.0 MB/s | 4.7 MB 00:00 2024-03-26T21:51:25.805 INFO:teuthology.orchestra.run.smithi194.stdout:(4/33): ceph-mds-18.2.2-983.gd83b58fa.el9.x86_6 2.7 MB/s | 2.1 MB 00:00 2024-03-26T21:51:25.830 INFO:teuthology.orchestra.run.smithi194.stdout:(5/33): ceph-mgr-modules-core-18.2.2-983.gd83b5 3.6 MB/s | 245 kB 00:00 2024-03-26T21:51:25.955 INFO:teuthology.orchestra.run.smithi194.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 310 kB/s | 32 kB 00:00 2024-03-26T21:51:25.981 INFO:teuthology.orchestra.run.smithi194.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 821 kB/s | 144 kB 00:00 2024-03-26T21:51:26.006 INFO:teuthology.orchestra.run.smithi194.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-26T21:51:26.031 INFO:teuthology.orchestra.run.smithi194.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-03-26T21:51:26.073 INFO:teuthology.orchestra.run.smithi194.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-26T21:51:26.115 INFO:teuthology.orchestra.run.smithi194.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-26T21:51:26.132 INFO:teuthology.orchestra.run.smithi194.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-26T21:51:26.157 INFO:teuthology.orchestra.run.smithi194.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-26T21:51:26.174 INFO:teuthology.orchestra.run.smithi194.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-26T21:51:26.358 INFO:teuthology.orchestra.run.smithi194.stdout:(15/33): ceph-osd-18.2.2-983.gd83b58fa.el9.x86_ 16 MB/s | 16 MB 00:01 2024-03-26T21:51:26.508 INFO:teuthology.orchestra.run.smithi194.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 356 kB/s | 125 kB 00:00 2024-03-26T21:51:26.567 INFO:teuthology.orchestra.run.smithi194.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-03-26T21:51:26.651 INFO:teuthology.orchestra.run.smithi194.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 177 kB/s | 11 kB 00:00 2024-03-26T21:51:26.675 INFO:teuthology.orchestra.run.smithi194.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 585 kB/s | 172 kB 00:00 2024-03-26T21:51:26.709 INFO:teuthology.orchestra.run.smithi194.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-03-26T21:51:26.734 INFO:teuthology.orchestra.run.smithi194.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-26T21:51:26.760 INFO:teuthology.orchestra.run.smithi194.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-26T21:51:26.801 INFO:teuthology.orchestra.run.smithi194.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-26T21:51:26.829 INFO:teuthology.orchestra.run.smithi194.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 691 kB/s | 46 kB 00:00 2024-03-26T21:51:26.852 INFO:teuthology.orchestra.run.smithi194.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-26T21:51:26.877 INFO:teuthology.orchestra.run.smithi194.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-26T21:51:26.903 INFO:teuthology.orchestra.run.smithi194.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-03-26T21:51:26.936 INFO:teuthology.orchestra.run.smithi194.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-26T21:51:26.953 INFO:teuthology.orchestra.run.smithi194.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-03-26T21:51:26.978 INFO:teuthology.orchestra.run.smithi194.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-26T21:51:27.028 INFO:teuthology.orchestra.run.smithi194.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-26T21:51:27.045 INFO:teuthology.orchestra.run.smithi194.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-26T21:51:27.070 INFO:teuthology.orchestra.run.smithi194.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-03-26T21:51:27.075 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:27.075 INFO:teuthology.orchestra.run.smithi194.stdout:Total 12 MB/s | 29 MB 00:02 2024-03-26T21:51:27.087 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:27.124 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:27.124 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:27.501 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:27.502 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:28.747 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:28.815 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-26T21:51:28.847 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-26T21:51:28.880 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-26T21:51:28.972 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-26T21:51:29.050 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-26T21:51:29.091 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-26T21:51:29.164 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-26T21:51:29.217 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-26T21:51:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 29/32 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-26T21:51:29.274 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-26T21:51:29.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-26T21:51:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-26T21:51:29.545 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-26T21:51:29.592 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-26T21:51:29.622 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-26T21:51:29.657 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-26T21:51:29.687 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-26T21:51:29.764 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-26T21:51:29.797 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T21:51:29.831 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-26T21:51:29.871 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-26T21:51:29.906 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-26T21:51:29.940 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-26T21:51:30.029 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-26T21:51:30.041 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 30/32 2024-03-26T21:51:30.066 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 30/32 2024-03-26T21:51:30.066 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:30.066 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T21:51:30.066 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:51:30.066 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:51:30.066 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:30.068 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T21:51:30.080 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-26T21:51:30.152 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-26T21:51:30.153 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T21:51:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-26T21:51:30.298 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-26T21:51:30.423 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T21:51:30.574 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-983.gd83b58fa.el9.n 28/33 2024-03-26T21:51:30.601 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 29/33 2024-03-26T21:51:30.626 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 29/33 2024-03-26T21:51:30.626 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:30.626 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T21:51:30.626 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:51:30.626 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:51:30.626 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:31.216 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T21:51:31.216 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 1/32 2024-03-26T21:51:31.216 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 2/32 2024-03-26T21:51:31.216 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 3/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 4/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 5/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 6/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 7/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:18.2.2-983.gd83b58fa.el9.x 8/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:18.2.2-983.gd83b58fa.el9.x86 9/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 10/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 11/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 12/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 13/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-26T21:51:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-26T21:51:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 29/32 2024-03-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 31/32 2024-03-26T21:51:32.035 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 30/33 2024-03-26T21:51:32.063 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 30/33 2024-03-26T21:51:32.063 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:32.063 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T21:51:32.063 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:51:32.063 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:51:32.063 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.271 INFO:teuthology.orchestra.run.smithi196.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T21:51:32.272 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:32.273 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:51:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 31/33 2024-03-26T21:51:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 31/33 2024-03-26T21:51:32.547 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:32.547 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T21:51:32.547 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:51:32.547 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:51:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:32.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-03-26T21:51:32.762 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 32/33 2024-03-26T21:51:32.785 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 32/33 2024-03-26T21:51:32.785 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:32.786 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T21:51:32.786 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:51:32.786 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:51:32.786 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:32.851 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 33/33 2024-03-26T21:51:33.170 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:27 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:51:33.337 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:51:33.338 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:51:33.338 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:33.338 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 40 M 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 166 k 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-26T21:51:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:33.340 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 41 M 2024-03-26T21:51:33.340 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 165 M 2024-03-26T21:51:33.340 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:51:33.568 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-26T21:51:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-26T21:51:33.651 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): libcephsqlite-18.2.2-983.gd83b58fa.el9.x 604 kB/s | 166 kB 00:00 2024-03-26T21:51:33.677 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-26T21:51:33.993 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 548 kB/s | 187 kB 00:00 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 33/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 1/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 2/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 3/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 4/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 5/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-983.gd83b58fa.el9.n 6/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-26T21:51:34.619 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-26T21:51:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-26T21:51:34.621 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-26T21:51:35.527 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-18.2.2-983.gd83b58fa.el9.x86_6 18 MB/s | 40 MB 00:02 2024-03-26T21:51:35.530 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:35.530 INFO:teuthology.orchestra.run.smithi196.stdout:Total 19 MB/s | 41 MB 00:02 2024-03-26T21:51:35.545 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:35.561 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T21:51:35.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T21:51:35.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:35.564 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:35.565 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:51:35.565 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:51:35.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-03-26T21:51:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:51:36.508 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:51 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:36.513 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T21:51:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T21:51:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:51:36.605 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:51:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T21:51:36.658 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:36.659 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:51:36.659 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 5/6 2024-03-26T21:51:36.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-03-26T21:51:37.285 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:52 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:37.433 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 224 k 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:51:37.434 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:37.435 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 224 k 2024-03-26T21:51:37.435 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 216 k 2024-03-26T21:51:37.435 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:37.843 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-18.2.2-983.gd83b58fa.el9.noarch.rpm 550 kB/s | 224 kB 00:00 2024-03-26T21:51:37.843 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:37.844 INFO:teuthology.orchestra.run.smithi194.stdout:Total 548 kB/s | 224 kB 00:00 2024-03-26T21:51:37.844 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:37.849 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:37.849 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:37.852 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:37.852 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:37.925 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:38.378 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:38.461 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:39.142 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:39.142 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:39.142 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:39.142 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:39.142 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:39.142 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:39.370 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T21:51:39.943 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:40.091 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 143 k 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:40.092 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:40.093 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:51:40.093 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:40.093 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 143 k 2024-03-26T21:51:40.093 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 395 k 2024-03-26T21:51:40.093 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 5/6 2024-03-26T21:51:40.438 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-18.2.2-983.gd83b58f 414 kB/s | 143 kB 00:00 2024-03-26T21:51:40.438 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:40.438 INFO:teuthology.orchestra.run.smithi194.stdout:Total 412 kB/s | 143 kB 00:00 2024-03-26T21:51:40.439 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:40.446 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:40.446 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 6/6 2024-03-26T21:51:40.498 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:40.498 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:40.633 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:40.713 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.e 1/1 2024-03-26T21:51:40.738 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.e 1/1 2024-03-26T21:51:40.738 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:40.738 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T21:51:40.738 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:41.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.e 1/1 2024-03-26T21:51:41.705 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:41.706 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:41.706 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:41.706 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:41.706 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:41.966 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-03-26T21:51:42.502 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 6/6 2024-03-26T21:51:42.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 1/6 2024-03-26T21:51:42.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 2/6 2024-03-26T21:51:42.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:51:42.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T21:51:42.502 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T21:51:42.536 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:42.637 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:51:42.688 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:42.689 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:51:42.689 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:42.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T21:51:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T21:51:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:42.874 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:42.875 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:51:43.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-03-26T21:51:43.316 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:43.465 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 1.8 M 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 23 k 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 15 k 2024-03-26T21:51:43.466 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout:Install 5 Packages 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:43.467 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.0 M 2024-03-26T21:51:43.468 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 8.6 M 2024-03-26T21:51:43.468 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:43.686 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:37 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:51:43.707 INFO:teuthology.orchestra.run.smithi194.stdout:(1/5): ceph-prometheus-alerts-18.2.2-983.gd83b5 77 kB/s | 15 kB 00:00 2024-03-26T21:51:43.847 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 6.5 k 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 2.1 M 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 1.4 M 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 245 k 2024-03-26T21:51:43.850 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 4.7 M 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 16 M 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T21:51:43.851 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T21:51:43.852 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout:Install 33 Packages 2024-03-26T21:51:43.853 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:43.854 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 29 M 2024-03-26T21:51:43.854 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 104 M 2024-03-26T21:51:43.854 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:51:43.891 INFO:teuthology.orchestra.run.smithi194.stdout:(2/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 168 kB/s | 31 kB 00:00 2024-03-26T21:51:44.025 INFO:teuthology.orchestra.run.smithi194.stdout:(3/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-26T21:51:44.083 INFO:teuthology.orchestra.run.smithi194.stdout:(4/5): ceph-mgr-dashboard-18.2.2-983.gd83b58fa. 3.1 MB/s | 1.8 MB 00:00 2024-03-26T21:51:44.271 INFO:teuthology.orchestra.run.smithi196.stdout:(1/33): ceph-18.2.2-983.gd83b58fa.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2024-03-26T21:51:44.679 INFO:teuthology.orchestra.run.smithi196.stdout:(2/33): ceph-mgr-18.2.2-983.gd83b58fa.el9.x86_6 2.5 MB/s | 1.4 MB 00:00 2024-03-26T21:51:44.800 INFO:teuthology.orchestra.run.smithi194.stdout:(5/5): ceph-grafana-dashboards-18.2.2-983.gd83b 18 kB/s | 23 kB 00:01 2024-03-26T21:51:44.803 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:44.804 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-26T21:51:44.804 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:44.811 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:44.811 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:44.871 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:44.871 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:45.080 INFO:teuthology.orchestra.run.smithi196.stdout:(3/33): ceph-mon-18.2.2-983.gd83b58fa.el9.x86_6 5.8 MB/s | 4.7 MB 00:00 2024-03-26T21:51:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:45.138 INFO:teuthology.orchestra.run.smithi196.stdout:(4/33): ceph-mgr-modules-core-18.2.2-983.gd83b5 4.1 MB/s | 245 kB 00:00 2024-03-26T21:51:45.205 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-26T21:51:45.238 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-26T21:51:45.266 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-983.gd83b58fa.el9.no 3/5 2024-03-26T21:51:45.280 INFO:teuthology.orchestra.run.smithi196.stdout:(5/33): ceph-mds-18.2.2-983.gd83b58fa.el9.x86_6 1.8 MB/s | 2.1 MB 00:01 2024-03-26T21:51:45.305 INFO:teuthology.orchestra.run.smithi196.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 864 kB/s | 144 kB 00:00 2024-03-26T21:51:45.323 INFO:teuthology.orchestra.run.smithi196.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-26T21:51:45.340 INFO:teuthology.orchestra.run.smithi196.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-26T21:51:45.356 INFO:teuthology.orchestra.run.smithi196.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-26T21:51:45.398 INFO:teuthology.orchestra.run.smithi196.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-26T21:51:45.440 INFO:teuthology.orchestra.run.smithi196.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-26T21:51:45.466 INFO:teuthology.orchestra.run.smithi196.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-26T21:51:45.491 INFO:teuthology.orchestra.run.smithi196.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T21:51:45.503 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-983.gd83b58fa.el9.n 4/5 2024-03-26T21:51:45.516 INFO:teuthology.orchestra.run.smithi196.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T21:51:45.528 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 5/5 2024-03-26T21:51:45.741 INFO:teuthology.orchestra.run.smithi196.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 500 kB/s | 125 kB 00:00 2024-03-26T21:51:45.925 INFO:teuthology.orchestra.run.smithi196.stdout:(16/33): ceph-osd-18.2.2-983.gd83b58fa.el9.x86_ 13 MB/s | 16 MB 00:01 2024-03-26T21:51:46.079 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 5/5 2024-03-26T21:51:46.079 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-983.gd83b58fa.el9.n 1/5 2024-03-26T21:51:46.079 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 2/5 2024-03-26T21:51:46.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-983.gd83b58fa.el9.no 3/5 2024-03-26T21:51:46.080 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-26T21:51:46.184 INFO:teuthology.orchestra.run.smithi196.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 65 kB/s | 43 kB 00:00 2024-03-26T21:51:46.284 INFO:teuthology.orchestra.run.smithi196.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-26T21:51:46.361 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:46.362 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:46.376 INFO:teuthology.orchestra.run.smithi196.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-26T21:51:46.468 INFO:teuthology.orchestra.run.smithi196.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-26T21:51:46.560 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T21:51:46.560 INFO:teuthology.orchestra.run.smithi196.stdout:(21/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 210 kB/s | 172 kB 00:00 2024-03-26T21:51:46.585 INFO:teuthology.orchestra.run.smithi196.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-03-26T21:51:46.652 INFO:teuthology.orchestra.run.smithi196.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-26T21:51:46.677 INFO:teuthology.orchestra.run.smithi196.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-26T21:51:46.761 INFO:teuthology.orchestra.run.smithi196.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-03-26T21:51:46.861 INFO:teuthology.orchestra.run.smithi196.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-26T21:51:46.895 INFO:teuthology.orchestra.run.smithi196.stdout:(27/33): python3-cherrypy-18.6.1-2.el9.noarch.r 379 kB/s | 358 kB 00:00 2024-03-26T21:51:46.995 INFO:teuthology.orchestra.run.smithi196.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-26T21:51:47.045 INFO:teuthology.orchestra.run.smithi196.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 489 kB/s | 90 kB 00:00 2024-03-26T21:51:47.104 INFO:teuthology.orchestra.run.smithi196.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 639 kB/s | 272 kB 00:00 2024-03-26T21:51:47.114 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:47.204 INFO:teuthology.orchestra.run.smithi196.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-26T21:51:47.265 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout:=========================================================================================== 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout:=========================================================================================== 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 7.4 M 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T21:51:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout:=========================================================================================== 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-03-26T21:51:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:47.268 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 37 M 2024-03-26T21:51:47.268 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 209 M 2024-03-26T21:51:47.268 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:47.371 INFO:teuthology.orchestra.run.smithi196.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 611 kB/s | 230 kB 00:00 2024-03-26T21:51:47.663 INFO:teuthology.orchestra.run.smithi196.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 692 kB/s | 427 kB 00:00 2024-03-26T21:51:47.669 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:47.670 INFO:teuthology.orchestra.run.smithi196.stdout:Total 7.6 MB/s | 29 MB 00:03 2024-03-26T21:51:47.683 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:51:47.722 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:51:47.723 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:51:47.822 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2024-03-26T21:51:47.864 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-26T21:51:47.939 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-26T21:51:47.964 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 345 kB/s | 34 kB 00:00 2024-03-26T21:51:48.089 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:51:48.089 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:51:48.123 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-26T21:51:48.190 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-26T21:51:48.215 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T21:51:48.424 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2024-03-26T21:51:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): ceph-mgr-diskprediction-local-18.2.2-98 8.8 MB/s | 7.4 MB 00:00 2024-03-26T21:51:48.675 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-26T21:51:48.699 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 710 kB/s | 195 kB 00:00 2024-03-26T21:51:48.741 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-26T21:51:48.744 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:48.744 INFO:teuthology.orchestra.run.smithi194.stdout:Total 25 MB/s | 37 MB 00:01 2024-03-26T21:51:48.784 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:48.807 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:48.807 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:49.238 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:49.239 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:49.333 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:51:49.391 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-26T21:51:49.431 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-26T21:51:49.464 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-26T21:51:49.556 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-26T21:51:49.634 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-26T21:51:49.675 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-26T21:51:49.754 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-26T21:51:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-26T21:51:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-26T21:51:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-26T21:51:49.960 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:49.962 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-26T21:51:50.049 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T21:51:50.108 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T21:51:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-26T21:51:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-26T21:51:50.206 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:51:50.224 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-26T21:51:50.257 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T21:51:50.258 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-26T21:51:50.289 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-26T21:51:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-26T21:51:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T21:51:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-26T21:51:50.436 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:51:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-26T21:51:50.473 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:51:50.508 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-26T21:51:50.542 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-26T21:51:50.550 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T21:51:50.630 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-26T21:51:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-26T21:51:50.745 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-26T21:51:50.890 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-26T21:51:51.015 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T21:51:51.177 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-983.gd83b58fa.el9.n 28/33 2024-03-26T21:51:51.203 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 29/33 2024-03-26T21:51:51.228 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 29/33 2024-03-26T21:51:51.228 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:51.228 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T21:51:51.228 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:51:51.228 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:51:51.228 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:51.230 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T21:51:51.327 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T21:51:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T21:51:52.697 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 30/33 2024-03-26T21:51:52.724 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 30/33 2024-03-26T21:51:52.724 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:52.724 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T21:51:52.724 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:51:52.724 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:51:52.724 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:53.173 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 31/33 2024-03-26T21:51:53.196 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 31/33 2024-03-26T21:51:53.196 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:53.196 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T21:51:53.196 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:51:53.196 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:51:53.196 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:53.398 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 32/33 2024-03-26T21:51:53.421 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 32/33 2024-03-26T21:51:53.422 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:51:53.422 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T21:51:53.422 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:51:53.422 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:51:53.422 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:53.486 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 33/33 2024-03-26T21:51:53.628 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T21:51:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58 12/12 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 33/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 1/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 2/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 3/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 4/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 5/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-983.gd83b58fa.el9.n 6/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-26T21:51:55.257 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-26T21:51:55.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-26T21:51:55.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-26T21:51:55.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-26T21:51:55.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-26T21:51:55.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-26T21:51:55.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-26T21:51:55.260 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-26T21:51:55.260 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58 12/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58 1/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T21:51:56.143 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T21:51:56.144 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T21:51:56.180 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-26T21:51:56.180 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:56.180 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:51:56.180 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T21:51:56.181 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T21:51:56.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T21:51:56.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T21:51:56.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T21:51:56.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T21:51:56.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T21:51:56.183 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:56.183 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:51:56.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T21:51:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:56.558 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:51:56.936 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-03-26T21:51:57.083 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:51 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:51:57.180 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:51:57.231 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:51:57.232 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:51:57.232 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:51:57.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-03-26T21:51:57.501 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:12 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:51:57.653 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 49 k 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T21:51:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-03-26T21:51:57.656 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-03-26T21:51:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:51:57.657 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.0 M 2024-03-26T21:51:57.657 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-03-26T21:51:57.657 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:51:57.851 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:51 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:51:57.895 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 176 kB/s | 24 kB 00:00 2024-03-26T21:51:57.920 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-03-26T21:51:57.979 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-26T21:51:57.999 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:51:57.999 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:57.999 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 224 k 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 224 k 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 216 k 2024-03-26T21:51:58.000 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:51:58.004 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-26T21:51:58.029 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T21:51:58.046 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-26T21:51:58.063 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): ceph-mgr-rook-18.2.2-983.gd83b58fa.el9. 161 kB/s | 49 kB 00:00 2024-03-26T21:51:58.205 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-03-26T21:51:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-26T21:51:58.247 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-18.2.2-983.gd83b58fa.el9.noarch.rpm 909 kB/s | 224 kB 00:00 2024-03-26T21:51:58.248 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:58.248 INFO:teuthology.orchestra.run.smithi196.stdout:Total 903 kB/s | 224 kB 00:00 2024-03-26T21:51:58.248 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:51:58.253 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:51:58.253 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:51:58.256 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:51:58.256 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:51:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:51:58.381 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 639 kB/s | 203 kB 00:00 2024-03-26T21:51:58.397 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-26T21:51:58.506 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 717 kB/s | 90 kB 00:00 2024-03-26T21:51:58.581 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-03-26T21:51:58.583 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:51:58.583 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-26T21:51:58.583 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:51:58.596 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:51:58.597 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:51:58.688 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:51:58.688 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:51:58.716 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:58.793 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:59.123 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:59.190 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:51:59.359 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T21:51:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/1 2024-03-26T21:51:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:59.406 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:51:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:51:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:51:59.406 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:51:59.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T21:51:59.470 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T21:51:59.500 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T21:51:59.541 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T21:51:59.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T21:51:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T21:51:59.752 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T21:51:59.801 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T21:52:00.176 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:00.323 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:00.324 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-26T21:52:00.324 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:00.324 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 143 k 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 143 k 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 395 k 2024-03-26T21:52:00.325 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:00.518 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T21:52:00.539 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-18.2.2-983.gd83b58f 668 kB/s | 143 kB 00:00 2024-03-26T21:52:00.539 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:00.540 INFO:teuthology.orchestra.run.smithi196.stdout:Total 663 kB/s | 143 kB 00:00 2024-03-26T21:52:00.540 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:00.547 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:00.547 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:00.564 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T21:52:00.598 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T21:52:00.599 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:00.599 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T21:52:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 13/13 2024-03-26T21:52:00.741 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.e 1/1 2024-03-26T21:52:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.e 1/1 2024-03-26T21:52:00.831 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:52:00.831 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T21:52:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:01.420 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 13/13 2024-03-26T21:52:01.421 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 1/13 2024-03-26T21:52:01.421 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T21:52:01.421 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T21:52:01.422 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T21:52:01.422 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T21:52:01.423 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T21:52:01.423 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T21:52:01.423 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T21:52:01.423 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T21:52:01.424 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T21:52:01.424 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T21:52:01.424 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T21:52:01.777 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T21:52:01.777 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:01.777 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:01.777 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:01.777 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:01.778 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:01.809 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.e 1/1 2024-03-26T21:52:01.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:01.810 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:01.810 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:01.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:01.810 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:02.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T21:52:02.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-03-26T21:52:02.609 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:17 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:02.637 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:02.767 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:02.767 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 138 k 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:02.768 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:02.769 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-03-26T21:52:02.769 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-03-26T21:52:02.769 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:02.769 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 6.8 M 2024-03-26T21:52:02.769 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 33 M 2024-03-26T21:52:02.769 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:02.788 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:02.789 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:02.789 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:02.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T21:52:03.406 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:03.557 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:03.557 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:03.557 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 1.8 M 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 23 k 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 15 k 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:03.558 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:03.559 INFO:teuthology.orchestra.run.smithi196.stdout:Install 5 Packages 2024-03-26T21:52:03.559 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:03.559 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-03-26T21:52:03.559 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 8.6 M 2024-03-26T21:52:03.559 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:03.586 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-26T21:52:03.657 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): ceph-mgr-cephadm-18.2.2-983.gd83b58fa.el 508 kB/s | 138 kB 00:00 2024-03-26T21:52:03.783 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-26T21:52:03.874 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 270 kB/s | 58 kB 00:00 2024-03-26T21:52:03.925 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-26T21:52:03.935 INFO:teuthology.orchestra.run.smithi196.stdout:(1/5): ceph-prometheus-alerts-18.2.2-983.gd83b5 78 kB/s | 15 kB 00:00 2024-03-26T21:52:03.950 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 307 kB/s | 51 kB 00:00 2024-03-26T21:52:03.952 INFO:teuthology.orchestra.run.smithi196.stdout:(2/5): ceph-grafana-dashboards-18.2.2-983.gd83b 112 kB/s | 23 kB 00:00 2024-03-26T21:52:03.953 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:03.953 INFO:teuthology.orchestra.run.smithi194.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-26T21:52:03.966 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:03.976 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:03.976 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:04.064 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:04.064 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:04.219 INFO:teuthology.orchestra.run.smithi196.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 108 kB/s | 31 kB 00:00 2024-03-26T21:52:04.372 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:04.394 INFO:teuthology.orchestra.run.smithi196.stdout:(4/5): ceph-mgr-dashboard-18.2.2-983.gd83b58fa. 2.7 MB/s | 1.8 MB 00:00 2024-03-26T21:52:04.469 INFO:teuthology.orchestra.run.smithi196.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 365 kB/s | 188 kB 00:00 2024-03-26T21:52:04.471 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:04.471 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-26T21:52:04.471 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:04.478 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:04.478 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:04.534 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T21:52:04.535 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:04.535 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:04.601 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T21:52:04.767 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:04.850 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-26T21:52:04.882 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-26T21:52:04.911 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-983.gd83b58fa.el9.no 3/5 2024-03-26T21:52:05.116 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T21:52:05.145 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-983.gd83b58fa.el9.n 4/5 2024-03-26T21:52:05.181 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 5/5 2024-03-26T21:52:05.195 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T21:52:05.257 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T21:52:05.280 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 6/6 2024-03-26T21:52:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 5/5 2024-03-26T21:52:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-983.gd83b58fa.el9.n 1/5 2024-03-26T21:52:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 2/5 2024-03-26T21:52:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-983.gd83b58fa.el9.no 3/5 2024-03-26T21:52:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-26T21:52:05.971 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-26T21:52:05.971 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:05.971 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:05.971 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:05.971 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:05.971 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:05.972 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T21:52:05.972 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T21:52:05.972 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:05.972 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:06.070 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 6/6 2024-03-26T21:52:06.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/6 2024-03-26T21:52:06.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T21:52:06.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T21:52:06.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T21:52:06.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T21:52:06.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:06.377 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:06.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-03-26T21:52:06.753 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:06.907 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================== 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================== 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 7.4 M 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T21:52:06.908 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================== 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-26T21:52:06.909 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:06.910 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-03-26T21:52:06.910 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-03-26T21:52:06.910 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:07.165 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:22 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 844 k 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-26T21:52:07.317 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 922 k 2024-03-26T21:52:07.318 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.7 M 2024-03-26T21:52:07.319 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:07.450 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 221 kB/s | 32 kB 00:00 2024-03-26T21:52:07.467 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 529 kB/s | 79 kB 00:00 2024-03-26T21:52:07.492 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-26T21:52:07.560 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T21:52:07.576 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 414 kB/s | 34 kB 00:00 2024-03-26T21:52:07.634 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-fuse-18.2.2-983.gd83b58fa.el9.x86_6 2.6 MB/s | 844 kB 00:00 2024-03-26T21:52:07.636 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:07.636 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.8 MB/s | 922 kB 00:00 2024-03-26T21:52:07.636 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:07.657 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:07.657 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:07.730 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:07.730 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:07.754 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-26T21:52:07.818 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-26T21:52:07.844 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T21:52:07.910 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:08.041 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T21:52:08.052 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2024-03-26T21:52:08.117 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 2/2 2024-03-26T21:52:08.270 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): ceph-mgr-diskprediction-local-18.2.2-98 7.7 MB/s | 7.4 MB 00:00 2024-03-26T21:52:08.303 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-26T21:52:08.328 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 776 kB/s | 195 kB 00:00 2024-03-26T21:52:08.420 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-26T21:52:08.423 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:08.424 INFO:teuthology.orchestra.run.smithi196.stdout:Total 24 MB/s | 37 MB 00:01 2024-03-26T21:52:08.461 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:08.483 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:08.483 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 2/2 2024-03-26T21:52:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/2 2024-03-26T21:52:08.892 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:08.892 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:09.093 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T21:52:09.093 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:09.093 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:09.093 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T21:52:09.093 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:09.093 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:09.346 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-03-26T21:52:09.594 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:09.667 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T21:52:09.699 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T21:52:09.791 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:52:09.827 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T21:52:09.920 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:24 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:10.006 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:52:10.043 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:52:10.075 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:10.075 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 127 k 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 127 k 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 456 k 2024-03-26T21:52:10.076 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:10.120 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T21:52:10.389 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-18.2.2-983.gd83b58fa.el9.x86_64. 406 kB/s | 127 kB 00:00 2024-03-26T21:52:10.390 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:10.390 INFO:teuthology.orchestra.run.smithi194.stdout:Total 404 kB/s | 127 kB 00:00 2024-03-26T21:52:10.390 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:10.395 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:10.395 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:10.450 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:10.450 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:10.602 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:10.711 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:10.787 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T21:52:10.914 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T21:52:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:11.410 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:11.410 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:11.410 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:11.410 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:11.410 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:11.410 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:11.639 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-03-26T21:52:12.157 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T21:52:12.216 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:12.315 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:12.370 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:12.372 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:12.372 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:12.441 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-03-26T21:52:13.011 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:28 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:13.161 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 31 k 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:13.162 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:13.163 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:52:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:13.163 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 31 k 2024-03-26T21:52:13.163 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 139 k 2024-03-26T21:52:13.163 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:13.182 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T21:52:13.209 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58 12/12 2024-03-26T21:52:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-18.2.2-983.gd83b58fa.el9.x86_64 182 kB/s | 31 kB 00:00 2024-03-26T21:52:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:13.334 INFO:teuthology.orchestra.run.smithi194.stdout:Total 181 kB/s | 31 kB 00:00 2024-03-26T21:52:13.334 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:13.337 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:13.337 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:13.361 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:13.361 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:13.450 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:13.557 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:14.200 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:14.200 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:14.376 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-03-26T21:52:14.937 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:15.033 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:15.085 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:15.086 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:15.086 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:15.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58 12/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58 1/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T21:52:15.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T21:52:15.635 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T21:52:15.635 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T21:52:15.699 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:15.794 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:15.846 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:15.847 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:15.847 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:15.912 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T21:52:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:16.056 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:16.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-03-26T21:52:16.467 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:16.565 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:16.618 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:16.619 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:16.620 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:16.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-03-26T21:52:16.958 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:10 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:17.103 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 49 k 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T21:52:17.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-03-26T21:52:17.106 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:17.107 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-03-26T21:52:17.107 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-26T21:52:17.107 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:17.254 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:17.284 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 182 kB/s | 24 kB 00:00 2024-03-26T21:52:17.310 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 113 kB/s | 18 kB 00:00 2024-03-26T21:52:17.352 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:17.368 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-26T21:52:17.393 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-26T21:52:17.404 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:17.405 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:17.405 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:17.419 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T21:52:17.436 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-26T21:52:17.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-03-26T21:52:17.570 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): ceph-mgr-rook-18.2.2-983.gd83b58fa.el9. 117 kB/s | 49 kB 00:00 2024-03-26T21:52:17.611 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 168 kB/s | 32 kB 00:00 2024-03-26T21:52:17.628 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-26T21:52:17.837 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-26T21:52:18.019 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:18.037 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 449 kB/s | 90 kB 00:00 2024-03-26T21:52:18.096 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 385 kB/s | 203 kB 00:00 2024-03-26T21:52:18.115 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:18.167 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:18.168 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:18.168 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:18.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-03-26T21:52:18.790 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:18.888 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 819 kB/s | 1.0 MB 00:01 2024-03-26T21:52:18.889 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:18.890 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:18.890 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-03-26T21:52:18.890 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:18.905 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:18.905 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:18.943 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:18.944 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-26T21:52:18.944 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:19.000 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:19.000 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:19.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-03-26T21:52:19.517 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:19.571 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T21:52:19.719 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 87 k 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 87 k 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 227 k 2024-03-26T21:52:19.720 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T21:52:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T21:52:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T21:52:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T21:52:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T21:52:20.041 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T21:52:20.090 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T21:52:20.127 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-18.2.2-983.gd83b58fa.el9.x86_64.rpm 213 kB/s | 87 kB 00:00 2024-03-26T21:52:20.128 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:20.128 INFO:teuthology.orchestra.run.smithi194.stdout:Total 212 kB/s | 87 kB 00:00 2024-03-26T21:52:20.128 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:20.135 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:20.135 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:20.190 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:20.191 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:20.335 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:20.442 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:20.803 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T21:52:20.834 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:20.854 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T21:52:20.905 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T21:52:20.944 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T21:52:20.966 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 13/13 2024-03-26T21:52:21.091 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:21.091 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:21.091 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:21.091 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:21.091 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:21.091 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:21.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 13/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 1/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T21:52:21.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T21:52:21.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T21:52:21.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T21:52:21.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T21:52:21.890 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:22.047 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 3.0 M 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.0 M 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 12 M 2024-03-26T21:52:22.048 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T21:52:22.115 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:22.116 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:22.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T21:52:22.906 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-18.2.2-983.gd83b58fa.el9.x86_64.rpm 3.5 MB/s | 3.0 MB 00:00 2024-03-26T21:52:22.907 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:22.907 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.5 MB/s | 3.0 MB 00:00 2024-03-26T21:52:22.908 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:22.909 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:16 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:22.918 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:22.918 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:22.991 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:22.992 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:23.061 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:23.062 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-03-26T21:52:23.062 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:52:23.062 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-03-26T21:52:23.062 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:23.062 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:18.2.2-983.gd83b58fa.el9 ceph-noarch 138 k 2024-03-26T21:52:23.062 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-26T21:52:23.063 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:23.064 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-03-26T21:52:23.064 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-03-26T21:52:23.064 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:23.408 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): ceph-mgr-cephadm-18.2.2-983.gd83b58fa.el 432 kB/s | 138 kB 00:00 2024-03-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 718 kB/s | 248 kB 00:00 2024-03-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-03-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-03-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-03-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T21:52:24.851 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-03-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 09:50:45 PM UTC. 2024-03-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 6/6 2024-03-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 172 k 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 172 k 2024-03-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 494 k 2024-03-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-26T21:52:25.801 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-18.2.2-983.gd83b58fa.el9.x86_64.rpm 762 kB/s | 172 kB 00:00 2024-03-26T21:52:25.802 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:25.802 INFO:teuthology.orchestra.run.smithi194.stdout:Total 756 kB/s | 172 kB 00:00 2024-03-26T21:52:25.802 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-26T21:52:25.809 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-26T21:52:25.810 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-26T21:52:25.865 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-26T21:52:25.865 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-26T21:52:26.010 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-26T21:52:26.124 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:26.358 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 6/6 2024-03-26T21:52:26.358 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 1/6 2024-03-26T21:52:26.358 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T21:52:26.358 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T21:52:26.358 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T21:52:26.358 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T21:52:26.584 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:18.2.2-983.gd83b58fa.el9.noarch 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:26.677 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:26.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:26.865 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:26.865 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-26T21:52:26.865 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:26.865 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:52:26.866 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-26T21:52:26.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-03-26T21:52:27.104 DEBUG:teuthology.parallel:result is None 2024-03-26T21:52:27.489 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:21 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:27.639 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 844 k 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-26T21:52:27.640 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 922 k 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.7 M 2024-03-26T21:52:27.641 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:27.785 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 548 kB/s | 79 kB 00:00 2024-03-26T21:52:28.010 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-18.2.2-983.gd83b58fa.el9.x86_6 2.2 MB/s | 844 kB 00:00 2024-03-26T21:52:28.011 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:28.011 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.4 MB/s | 922 kB 00:00 2024-03-26T21:52:28.012 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:28.032 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:28.032 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:28.103 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:28.104 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:28.425 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T21:52:28.501 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 2/2 2024-03-26T21:52:29.237 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 2/2 2024-03-26T21:52:29.238 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/2 2024-03-26T21:52:29.551 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T21:52:29.551 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:29.551 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:29.552 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T21:52:29.552 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:29.552 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:29.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-03-26T21:52:30.399 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:24 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:30.548 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 127 k 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:30.549 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-03-26T21:52:30.550 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-03-26T21:52:30.550 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:30.775 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-18.2.2-983.gd83b58fa.el9.x86_64. 563 kB/s | 127 kB 00:00 2024-03-26T21:52:30.775 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:30.775 INFO:teuthology.orchestra.run.smithi196.stdout:Total 560 kB/s | 127 kB 00:00 2024-03-26T21:52:30.775 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:30.781 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:30.781 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:30.835 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:30.836 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:31.004 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:31.098 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:31.542 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:31.845 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:31.845 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:32.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-03-26T21:52:32.685 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:32.784 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:32.839 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:32.840 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:32.840 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:32.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-03-26T21:52:33.476 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:33.624 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:33.625 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:33.625 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 31 k 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 139 k 2024-03-26T21:52:33.626 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:33.787 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-18.2.2-983.gd83b58fa.el9.x86_64 193 kB/s | 31 kB 00:00 2024-03-26T21:52:33.788 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:33.788 INFO:teuthology.orchestra.run.smithi196.stdout:Total 191 kB/s | 31 kB 00:00 2024-03-26T21:52:33.788 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:33.791 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:33.791 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:33.820 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:33.821 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:33.915 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:34.018 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:34.379 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:34.633 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:34.633 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:34.633 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:34.633 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:34.633 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:34.633 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:34.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-03-26T21:52:35.393 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:35.490 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:35.542 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:35.543 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:35.543 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:35.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-03-26T21:52:36.178 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:36.273 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:36.325 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:36.326 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:36.326 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:36.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-03-26T21:52:36.945 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:37.041 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:37.094 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:37.095 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:37.095 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:37.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-03-26T21:52:37.717 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:37.814 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:37.866 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:37.867 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:37.867 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:37.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-03-26T21:52:38.491 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:38.587 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:38.639 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:38.640 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:38.641 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:38.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-03-26T21:52:39.265 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:39.362 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:18.2.2-983.gd83b58fa.el9.x86_64 is already installed. 2024-03-26T21:52:39.415 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:39.416 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-26T21:52:39.416 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:39.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-03-26T21:52:40.037 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:34 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:40.187 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 87 k 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 87 k 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 227 k 2024-03-26T21:52:40.188 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:40.498 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-18.2.2-983.gd83b58fa.el9.x86_64.rpm 280 kB/s | 87 kB 00:00 2024-03-26T21:52:40.499 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:40.499 INFO:teuthology.orchestra.run.smithi196.stdout:Total 279 kB/s | 87 kB 00:00 2024-03-26T21:52:40.499 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:40.506 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:40.506 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:40.561 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:40.561 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:40.702 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:40.805 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:41.201 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:41.482 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:41.482 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:41.482 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:41.482 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:41.483 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:41.483 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:41.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-03-26T21:52:42.293 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:42.445 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 3.0 M 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.0 M 2024-03-26T21:52:42.446 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 12 M 2024-03-26T21:52:42.447 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:43.059 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-18.2.2-983.gd83b58fa.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-26T21:52:43.060 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:43.060 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.9 MB/s | 3.0 MB 00:00 2024-03-26T21:52:43.060 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:43.069 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:43.069 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:43.141 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:43.141 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:43.546 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:43.593 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:43.616 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:43.617 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:52:43.617 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T21:52:43.617 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:52:43.617 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:52:43.617 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:44.716 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:44.716 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:44.716 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:44.716 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:44.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-03-26T21:52:45.522 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 09:51:06 PM UTC. 2024-03-26T21:52:45.673 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-26T21:52:45.673 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-26T21:52:45.673 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:45.673 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-26T21:52:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:18.2.2-983.gd83b58fa.el9 ceph 172 k 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 172 k 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-03-26T21:52:45.674 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-26T21:52:45.880 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-18.2.2-983.gd83b58fa.el9.x86_64.rpm 839 kB/s | 172 kB 00:00 2024-03-26T21:52:45.880 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:52:45.880 INFO:teuthology.orchestra.run.smithi196.stdout:Total 833 kB/s | 172 kB 00:00 2024-03-26T21:52:45.880 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-26T21:52:45.888 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-26T21:52:45.888 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-26T21:52:45.944 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-26T21:52:45.944 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-26T21:52:46.092 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-26T21:52:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:46.613 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:46.894 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 1/1 2024-03-26T21:52:46.894 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:46.894 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-26T21:52:46.894 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:18.2.2-983.gd83b58fa.el9.x86_64 2024-03-26T21:52:46.895 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:52:46.895 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-26T21:52:47.124 DEBUG:teuthology.parallel:result is None 2024-03-26T21:52:47.125 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:52:47.125 INFO:teuthology.packaging:ref: None 2024-03-26T21:52:47.125 INFO:teuthology.packaging:tag: None 2024-03-26T21:52:47.125 INFO:teuthology.packaging:branch: reef 2024-03-26T21:52:47.125 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:52:47.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-26T21:52:47.469 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi194.stdout:18.2.2-983.gd83b58fa.el9 2024-03-26T21:52:47.495 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-983.gd83b58fa.el9 2024-03-26T21:52:47.495 INFO:teuthology.task.install:The correct ceph version 18.2.2-983.gd83b58fa is installed. 2024-03-26T21:52:47.497 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:52:47.497 INFO:teuthology.packaging:ref: None 2024-03-26T21:52:47.497 INFO:teuthology.packaging:tag: None 2024-03-26T21:52:47.497 INFO:teuthology.packaging:branch: reef 2024-03-26T21:52:47.497 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:52:47.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-26T21:52:47.745 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T21:52:47.767 INFO:teuthology.orchestra.run.smithi196.stdout:18.2.2-983.gd83b58fa.el9 2024-03-26T21:52:47.767 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-983.gd83b58fa.el9 2024-03-26T21:52:47.767 INFO:teuthology.task.install:The correct ceph version 18.2.2-983.gd83b58fa is installed. 2024-03-26T21:52:47.769 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T21:52:47.769 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:52:47.769 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T21:52:47.802 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:52:47.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T21:52:47.835 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T21:52:47.835 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:52:47.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T21:52:47.872 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T21:52:47.940 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:52:47.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T21:52:47.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T21:52:48.033 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T21:52:48.033 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:52:48.034 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T21:52:48.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T21:52:48.132 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:52:48.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T21:52:48.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T21:52:48.225 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T21:52:48.225 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:52:48.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T21:52:48.252 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T21:52:48.319 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:52:48.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T21:52:48.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T21:52:48.413 INFO:teuthology.run_tasks:Running task print... 2024-03-26T21:52:48.422 INFO:teuthology.task.print:**** done install task... 2024-03-26T21:52:48.422 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T21:52:48.519 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'} 2024-03-26T21:52:48.519 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-03-26T21:52:48.520 INFO:tasks.cephadm:Cluster fsid is 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:52:48.520 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T21:52:48.520 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-26T21:52:48.520 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi194': '172.21.15.194', 'mon.smithi196': '172.21.15.196'} 2024-03-26T21:52:48.520 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T21:52:48.520 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-03-26T21:52:48.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-03-26T21:52:48.575 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-03-26T21:52:48.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:52:48.789 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77974/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2484-g42fc52b8', 'node_name': '172.21.4.61+sulcata01', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2484.g42fc52b8'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-26 18:30:27.407480', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-2024-03-26-0925', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-03-26T21:52:49.128 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 d83b58fabace7c0b0c2d45a3ac2899524ef89e97 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-03-26T21:52:49.130 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/x86_64/flavors/default/cephadm 2024-03-26T21:52:49.130 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T21:52:49.548 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 215331 Mar 26 21:52 /home/ubuntu/cephtest/cephadm 2024-03-26T21:52:49.549 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/d83b58fabace7c0b0c2d45a3ac2899524ef89e97/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T21:52:49.962 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 215331 Mar 26 21:52 /home/ubuntu/cephtest/cephadm 2024-03-26T21:52:49.963 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T21:52:49.979 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T21:52:50.007 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-03-26T21:52:50.007 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-26T21:52:50.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-26T21:52:50.766 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-26T21:52:50.832 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 18.2.2-983-gd83b58fa (d83b58fabace7c0b0c2d45a3ac2899524ef89e97) reef (stable)", 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "4c6f72a25bfc8f895430ebfdfa5b9e4c4e63b2b11225a224873d069d9b227ce1", 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:146923a4e32dab24386c625907ce68ea89537d78fd84a315a9e4bb72be8d7740" 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-03-26T21:53:22.803 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-26T21:53:39.942 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-26T21:53:39.943 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 18.2.2-983-gd83b58fa (d83b58fabace7c0b0c2d45a3ac2899524ef89e97) reef (stable)", 2024-03-26T21:53:39.943 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "4c6f72a25bfc8f895430ebfdfa5b9e4c4e63b2b11225a224873d069d9b227ce1", 2024-03-26T21:53:39.943 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-03-26T21:53:39.943 INFO:teuthology.orchestra.run.smithi194.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:146923a4e32dab24386c625907ce68ea89537d78fd84a315a9e4bb72be8d7740" 2024-03-26T21:53:39.943 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-03-26T21:53:39.943 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-26T21:53:39.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-03-26T21:53:40.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-03-26T21:53:40.055 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-03-26T21:53:40.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-03-26T21:53:40.124 INFO:tasks.cephadm:Writing seed config... 2024-03-26T21:53:40.125 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-26T21:53:40.126 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-26T21:53:40.127 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T21:53:40.128 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T21:53:40.129 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T21:53:40.129 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-26T21:53:40.129 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-26T21:53:40.129 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:53:40.130 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T21:53:40.144 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2f9b9224-ebbb-11ee-95d0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-26T21:53:40.144 DEBUG:teuthology.orchestra.run.smithi194:mon.smithi194> sudo journalctl -f -n 0 -u ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194.service 2024-03-26T21:53:40.187 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T21:53:40.187 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.194 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:53:40.342 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:53:40.343 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '2f9b9224-ebbb-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.194', '--skip-admin-label'] 2024-03-26T21:53:40.369 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:53:40.370 INFO:teuthology.orchestra.run.smithi194.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T21:53:40.370 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying podman|docker is present... 2024-03-26T21:53:40.391 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:53:40.391 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying lvm2 is present... 2024-03-26T21:53:40.391 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying time synchronization is in place... 2024-03-26T21:53:40.398 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T21:53:40.398 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T21:53:40.404 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T21:53:40.404 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout inactive 2024-03-26T21:53:40.410 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout enabled 2024-03-26T21:53:40.416 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout active 2024-03-26T21:53:40.416 INFO:teuthology.orchestra.run.smithi194.stdout:Unit chronyd.service is enabled and running 2024-03-26T21:53:40.416 INFO:teuthology.orchestra.run.smithi194.stdout:Repeating the final host check... 2024-03-26T21:53:40.439 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:53:40.439 INFO:teuthology.orchestra.run.smithi194.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T21:53:40.440 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl is present 2024-03-26T21:53:40.440 INFO:teuthology.orchestra.run.smithi194.stdout:lvcreate is present 2024-03-26T21:53:40.446 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T21:53:40.446 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T21:53:40.451 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T21:53:40.451 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout inactive 2024-03-26T21:53:40.457 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout enabled 2024-03-26T21:53:40.463 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout active 2024-03-26T21:53:40.463 INFO:teuthology.orchestra.run.smithi194.stdout:Unit chronyd.service is enabled and running 2024-03-26T21:53:40.463 INFO:teuthology.orchestra.run.smithi194.stdout:Host looks OK 2024-03-26T21:53:40.463 INFO:teuthology.orchestra.run.smithi194.stdout:Cluster fsid: 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:40.463 INFO:teuthology.orchestra.run.smithi194.stdout:Acquiring lock 140131955528032 on /run/cephadm/2f9b9224-ebbb-11ee-95d0-87774f69a715.lock 2024-03-26T21:53:40.464 INFO:teuthology.orchestra.run.smithi194.stdout:Lock 140131955528032 acquired on /run/cephadm/2f9b9224-ebbb-11ee-95d0-87774f69a715.lock 2024-03-26T21:53:40.464 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying IP 172.21.15.194 port 3300 ... 2024-03-26T21:53:40.464 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying IP 172.21.15.194 port 6789 ... 2024-03-26T21:53:40.464 INFO:teuthology.orchestra.run.smithi194.stdout:Base mon IP(s) is [172.21.15.194:3300, 172.21.15.194:6789], mon addrv is [v2:172.21.15.194:3300,v1:172.21.15.194:6789] 2024-03-26T21:53:40.467 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.194 metric 100 2024-03-26T21:53:40.467 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.194 metric 100 2024-03-26T21:53:40.470 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T21:53:40.470 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T21:53:40.470 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-03-26T21:53:40.472 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T21:53:40.472 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T21:53:40.472 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T21:53:40.472 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T21:53:40.472 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7ea5/64 scope link 2024-03-26T21:53:40.472 INFO:teuthology.orchestra.run.smithi194.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T21:53:40.473 INFO:teuthology.orchestra.run.smithi194.stdout:Mon IP `172.21.15.194` is in CIDR network `172.21.0.0/20` 2024-03-26T21:53:40.473 INFO:teuthology.orchestra.run.smithi194.stdout:Mon IP `172.21.15.194` is in CIDR network `172.21.0.0/20` 2024-03-26T21:53:40.473 INFO:teuthology.orchestra.run.smithi194.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T21:53:40.474 INFO:teuthology.orchestra.run.smithi194.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T21:53:40.474 INFO:teuthology.orchestra.run.smithi194.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stdout 4c6f72a25bfc8f895430ebfdfa5b9e4c4e63b2b11225a224873d069d9b227ce1 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stderr Copying blob sha256:337750336ad7a9c713ab956443b43046a8799fc7a5fcbbf00e4fed5e39c8c336 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stderr Copying config sha256:4c6f72a25bfc8f895430ebfdfa5b9e4c4e63b2b11225a224873d069d9b227ce1 2024-03-26T21:53:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T21:53:41.949 INFO:teuthology.orchestra.run.smithi194.stdout:ceph: stdout ceph version 18.2.2-983-gd83b58fa (d83b58fabace7c0b0c2d45a3ac2899524ef89e97) reef (stable) 2024-03-26T21:53:41.949 INFO:teuthology.orchestra.run.smithi194.stdout:Ceph version: ceph version 18.2.2-983-gd83b58fa (d83b58fabace7c0b0c2d45a3ac2899524ef89e97) reef (stable) 2024-03-26T21:53:41.949 INFO:teuthology.orchestra.run.smithi194.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T21:53:42.771 INFO:teuthology.orchestra.run.smithi194.stdout:stat: stdout 167 167 2024-03-26T21:53:42.771 INFO:teuthology.orchestra.run.smithi194.stdout:Creating initial keys... 2024-03-26T21:53:43.651 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph-authtool: stdout AQDnQwNm+FUNBxAAvAbCfozxkA/K8xQkG9xSKw== 2024-03-26T21:53:44.523 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph-authtool: stdout AQDnQwNmmh4WORAANX0AMJkhgo5gLoqTugTo6g== 2024-03-26T21:53:45.496 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph-authtool: stdout AQDoQwNmlL7xMxAAsaIAvvXTSC3X6IefcNq7ag== 2024-03-26T21:53:45.498 INFO:teuthology.orchestra.run.smithi194.stdout:Creating initial monmap... 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:monmaptool for smithi194 [v2:172.21.15.194:3300,v1:172.21.15.194:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:setting min_mon_release = pacific 2024-03-26T21:53:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/monmaptool: set fsid to 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:46.370 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T21:53:46.370 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:53:46.370 INFO:teuthology.orchestra.run.smithi194.stdout:Creating mon... 2024-03-26T21:53:47.408 INFO:teuthology.orchestra.run.smithi194.stdout:create mon.smithi194 on 2024-03-26T21:53:47.605 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T21:53:47.741 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T21:53:47.903 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715.target. 2024-03-26T21:53:47.903 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715.target. 2024-03-26T21:53:48.128 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194 2024-03-26T21:53:48.128 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194.service: Unit ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194.service not loaded. 2024-03-26T21:53:48.297 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715.target.wants/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194.service → /etc/systemd/system/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@.service. 2024-03-26T21:53:48.848 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:53:48.848 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout disabled 2024-03-26T21:53:48.855 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:53:48.855 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout inactive 2024-03-26T21:53:48.855 INFO:teuthology.orchestra.run.smithi194.stdout:firewalld.service is not enabled 2024-03-26T21:53:48.855 INFO:teuthology.orchestra.run.smithi194.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T21:53:48.855 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for mon to start... 2024-03-26T21:53:48.855 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for mon... 2024-03-26T21:53:48.980 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:48 smithi194 systemd[1]: Started Ceph mon.smithi194 for 2f9b9224-ebbb-11ee-95d0-87774f69a715. 2024-03-26T21:53:49.910 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:49 smithi194 ceph-mon[29317]: mon.smithi194 is new leader, mons smithi194 in quorum (ranks 0) 2024-03-26T21:53:49.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:49 smithi194 ceph-mon[29317]: monmap e1: 1 mons at {smithi194=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T21:53:49.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:49 smithi194 ceph-mon[29317]: fsmap 2024-03-26T21:53:49.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:49 smithi194 ceph-mon[29317]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T21:53:49.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:49 smithi194 ceph-mon[29317]: mgrmap e1: no daemons active 2024-03-26T21:53:49.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:49 smithi194 ceph-mon[29317]: from='client.? 172.21.15.194:0/2670665495' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T21:53:49.934 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T21:53:49.934 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout id: 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:49.934 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T21:53:49.934 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout services: 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi194 (age 0.516932s) 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout data: 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:53:49.935 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55ca77700 1 Processor -- start 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55ca77700 1 -- start start 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55ca77700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff558107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55ca77700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581084e0 con 0x7ff558107af0 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff558107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff558107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48602/0 (socket says 172.21.15.194:48602) 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.266+0000 7ff55659c700 1 -- 172.21.15.194:0/1949861742 learned_addr learned my addr 172.21.15.194:0/1949861742 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.267+0000 7ff55659c700 1 -- 172.21.15.194:0/1949861742 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff558108620 con 0x7ff558107af0 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.267+0000 7ff55659c700 1 --2- 172.21.15.194:0/1949861742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff558107f10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff5480098a0 tx=0x7ff548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c125c50028e0ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.267+0000 7ff55559a700 1 -- 172.21.15.194:0/1949861742 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff548004180 con 0x7ff558107af0 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.267+0000 7ff55559a700 1 -- 172.21.15.194:0/1949861742 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5480042e0 con 0x7ff558107af0 2024-03-26T21:53:49.936 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.267+0000 7ff55559a700 1 -- 172.21.15.194:0/1949861742 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5480036a0 con 0x7ff558107af0 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 -- 172.21.15.194:0/1949861742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 msgr2=0x7ff558107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 --2- 172.21.15.194:0/1949861742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff558107f10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff5480098a0 tx=0x7ff548009bb0 comp rx=0 tx=0).stop 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 -- 172.21.15.194:0/1949861742 shutdown_connections 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 --2- 172.21.15.194:0/1949861742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff558107f10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 -- 172.21.15.194:0/1949861742 >> 172.21.15.194:0/1949861742 conn(0x7ff558100110 msgr2=0x7ff558102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 -- 172.21.15.194:0/1949861742 shutdown_connections 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.268+0000 7ff55ca77700 1 -- 172.21.15.194:0/1949861742 wait complete. 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55ca77700 1 Processor -- start 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55ca77700 1 -- start start 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55ca77700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff5581999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55ca77700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558199f00 con 0x7ff558107af0 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff5581999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:53:49.937 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff5581999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48604/0 (socket says 172.21.15.194:48604) 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55659c700 1 -- 172.21.15.194:0/2670665495 learned_addr learned my addr 172.21.15.194:0/2670665495 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.269+0000 7ff55659c700 1 -- 172.21.15.194:0/2670665495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff548009550 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff55659c700 1 --2- 172.21.15.194:0/2670665495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff5581999c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff548004000 tx=0x7ff548004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff548003ce0 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548020070 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55819a100 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff548011630 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55819a5a0 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff548022030 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.270+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff548025d10 con 0x7ff558107af0 2024-03-26T21:53:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.271+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff558193770 con 0x7ff558107af0 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.275+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ff548025480 con 0x7ff558107af0 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.346+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7ff5581048c0 con 0x7ff558107af0 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.347+0000 7ff5477fe700 1 -- 172.21.15.194:0/2670665495 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7ff548011c20 con 0x7ff558107af0 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 msgr2=0x7ff5581999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 --2- 172.21.15.194:0/2670665495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff5581999c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff548004000 tx=0x7ff548004750 comp rx=0 tx=0).stop 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 shutdown_connections 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 --2- 172.21.15.194:0/2670665495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff558107af0 0x7ff5581999c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 >> 172.21.15.194:0/2670665495 conn(0x7ff558100110 msgr2=0x7ff5581908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 shutdown_connections 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:49.348+0000 7ff55ca77700 1 -- 172.21.15.194:0/2670665495 wait complete. 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:mon is available 2024-03-26T21:53:49.939 INFO:teuthology.orchestra.run.smithi194.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T21:53:50.981 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:53:50.981 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T21:53:50.981 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout fsid = 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.194:3300,v1:172.21.15.194:6789] 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-26T21:53:50.982 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c46a1700 1 Processor -- start 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c46a1700 1 -- start start 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c46a1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c46a1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc106f00 con 0x7f50bc106510 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c243d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c243d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48610/0 (socket says 172.21.15.194:48610) 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.404+0000 7f50c243d700 1 -- 172.21.15.194:0/3488272797 learned_addr learned my addr 172.21.15.194:0/3488272797 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.405+0000 7f50c243d700 1 -- 172.21.15.194:0/3488272797 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc1076c0 con 0x7f50bc106510 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.405+0000 7f50c243d700 1 --2- 172.21.15.194:0/3488272797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc106930 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f50b80098a0 tx=0x7f50b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fb8efcc0e48b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.405+0000 7f50c143b700 1 -- 172.21.15.194:0/3488272797 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b8004180 con 0x7f50bc106510 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.405+0000 7f50c143b700 1 -- 172.21.15.194:0/3488272797 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b80042e0 con 0x7f50bc106510 2024-03-26T21:53:50.983 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.405+0000 7f50c143b700 1 -- 172.21.15.194:0/3488272797 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b80036a0 con 0x7f50bc106510 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- 172.21.15.194:0/3488272797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 msgr2=0x7f50bc106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 --2- 172.21.15.194:0/3488272797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc106930 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f50b80098a0 tx=0x7f50b8009bb0 comp rx=0 tx=0).stop 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- 172.21.15.194:0/3488272797 shutdown_connections 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 --2- 172.21.15.194:0/3488272797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc106930 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- 172.21.15.194:0/3488272797 >> 172.21.15.194:0/3488272797 conn(0x7f50bc101b10 msgr2=0x7f50bc103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- 172.21.15.194:0/3488272797 shutdown_connections 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- 172.21.15.194:0/3488272797 wait complete. 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 Processor -- start 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- start start 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc07bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.406+0000 7f50c46a1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc07c040 con 0x7f50bc106510 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c243d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc07bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c243d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc07bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48626/0 (socket says 172.21.15.194:48626) 2024-03-26T21:53:50.984 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c243d700 1 -- 172.21.15.194:0/842065889 learned_addr learned my addr 172.21.15.194:0/842065889 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c243d700 1 -- 172.21.15.194:0/842065889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8009550 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c243d700 1 --2- 172.21.15.194:0/842065889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc07bb00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f50b8004000 tx=0x7f50b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b8003ce0 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b8020070 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc07a280 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b8011630 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc07a720 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f50b8022030 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.407+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f50b8025d40 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.408+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc0623e0 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.410+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f50b8025480 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.477+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f50bc107a30 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.480+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f50b8011c20 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.480+0000 7f50b37fe700 1 -- 172.21.15.194:0/842065889 <== mon.0 v2:172.21.15.194:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f50b8037590 con 0x7f50bc106510 2024-03-26T21:53:50.985 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.481+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 msgr2=0x7f50bc07bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.481+0000 7f50c46a1700 1 --2- 172.21.15.194:0/842065889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc07bb00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f50b8004000 tx=0x7f50b8004750 comp rx=0 tx=0).stop 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.482+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 shutdown_connections 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.482+0000 7f50c46a1700 1 --2- 172.21.15.194:0/842065889 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50bc106510 0x7f50bc07bb00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.482+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 >> 172.21.15.194:0/842065889 conn(0x7f50bc101b10 msgr2=0x7f50bc190820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.482+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 shutdown_connections 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:50.482+0000 7f50c46a1700 1 -- 172.21.15.194:0/842065889 wait complete. 2024-03-26T21:53:50.986 INFO:teuthology.orchestra.run.smithi194.stdout:Generating new minimal ceph.conf... 2024-03-26T21:53:51.080 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:50 smithi194 ceph-mon[29317]: from='client.? 172.21.15.194:0/842065889' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T21:53:51.080 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:50 smithi194 ceph-mon[29317]: from='client.? 172.21.15.194:0/842065889' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T21:53:52.160 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:51 smithi194 ceph-mon[29317]: from='client.? 172.21.15.194:0/310387502' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:57.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b8e99700 1 Processor -- start 2024-03-26T21:53:57.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b8e99700 1 -- start start 2024-03-26T21:53:57.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b8e99700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b41093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:53:57.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b8e99700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4079e50 con 0x7f83b4106fb0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b41093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b41093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48642/0 (socket says 172.21.15.194:48642) 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b259c700 1 -- 172.21.15.194:0/636741193 learned_addr learned my addr 172.21.15.194:0/636741193 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b259c700 1 -- 172.21.15.194:0/636741193 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b41098e0 con 0x7f83b4106fb0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.359+0000 7f83b259c700 1 --2- 172.21.15.194:0/636741193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b41093a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f83a40098a0 tx=0x7f83a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13ca43543919af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b159a700 1 -- 172.21.15.194:0/636741193 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4004180 con 0x7f83b4106fb0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b159a700 1 -- 172.21.15.194:0/636741193 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f83a40042e0 con 0x7f83b4106fb0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b159a700 1 -- 172.21.15.194:0/636741193 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4011a70 con 0x7f83b4106fb0 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 -- 172.21.15.194:0/636741193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 msgr2=0x7f83b41093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 --2- 172.21.15.194:0/636741193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b41093a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f83a40098a0 tx=0x7f83a4009bb0 comp rx=0 tx=0).stop 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 -- 172.21.15.194:0/636741193 shutdown_connections 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 --2- 172.21.15.194:0/636741193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b41093a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:53:57.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 -- 172.21.15.194:0/636741193 >> 172.21.15.194:0/636741193 conn(0x7f83b4100a00 msgr2=0x7f83b4102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 -- 172.21.15.194:0/636741193 shutdown_connections 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.360+0000 7f83b8e99700 1 -- 172.21.15.194:0/636741193 wait complete. 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.361+0000 7f83b8e99700 1 Processor -- start 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.361+0000 7f83b8e99700 1 -- start start 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.361+0000 7f83b8e99700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.361+0000 7f83b8e99700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b419a750 con 0x7f83b4106fb0 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48648/0 (socket says 172.21.15.194:48648) 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b259c700 1 -- 172.21.15.194:0/310387502 learned_addr learned my addr 172.21.15.194:0/310387502 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b259c700 1 -- 172.21.15.194:0/310387502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4009550 con 0x7f83b4106fb0 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b259c700 1 --2- 172.21.15.194:0/310387502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b419a210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83a4011710 tx=0x7f83a40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:53:57.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4003b10 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f83a401b8c0 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a40197e0 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b419a950 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b419adf0 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f83a4019cd0 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.362+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f83a401a710 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.363+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b410a630 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.365+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f83a4029060 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.431+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f83b410a7f0 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.431+0000 7f83a37fe700 1 -- 172.21.15.194:0/310387502 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f83a401f070 con 0x7f83b4106fb0 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 msgr2=0x7f83b419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 --2- 172.21.15.194:0/310387502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b419a210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83a4011710 tx=0x7f83a40047c0 comp rx=0 tx=0).stop 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 shutdown_connections 2024-03-26T21:53:57.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 --2- 172.21.15.194:0/310387502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83b4106fb0 0x7f83b419a210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:53:57.273 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 >> 172.21.15.194:0/310387502 conn(0x7f83b4100a00 msgr2=0x7f83b4101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:53:57.273 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 shutdown_connections 2024-03-26T21:53:57.273 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:51.432+0000 7f83b8e99700 1 -- 172.21.15.194:0/310387502 wait complete. 2024-03-26T21:53:57.273 INFO:teuthology.orchestra.run.smithi194.stdout:Restarting the monitor... 2024-03-26T21:53:57.532 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:57 smithi194 systemd[1]: Stopping Ceph mon.smithi194 for 2f9b9224-ebbb-11ee-95d0-87774f69a715... 2024-03-26T21:53:57.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:57 smithi194 ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194[29313]: 2024-03-26T21:53:57.536+0000 7f19c1585700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi194 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:53:57.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:57 smithi194 ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194[29313]: 2024-03-26T21:53:57.536+0000 7f19c1585700 -1 mon.smithi194@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T21:53:57.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:57 smithi194 podman[29566]: 2024-03-26 21:53:57.772617958 +0000 UTC m=+0.336711078 container died bc5abb7b98c93aada627ca668f0fd47129c9715f1572778849df28dab56732fa (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, RELEASE=reef-d83b58f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T21:53:58.398 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 podman[29566]: 2024-03-26 21:53:58.119625392 +0000 UTC m=+0.683718502 container cleanup bc5abb7b98c93aada627ca668f0fd47129c9715f1572778849df28dab56732fa (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-d83b58f) 2024-03-26T21:53:58.398 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 bash[29566]: ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194 2024-03-26T21:53:58.660 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 podman[29580]: 2024-03-26 21:53:58.403835789 +0000 UTC m=+0.629900973 container remove bc5abb7b98c93aada627ca668f0fd47129c9715f1572778849df28dab56732fa (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-d83b58f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T21:53:58.660 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 systemd[1]: ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194.service: Deactivated successfully. 2024-03-26T21:53:58.661 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 systemd[1]: Stopped Ceph mon.smithi194 for 2f9b9224-ebbb-11ee-95d0-87774f69a715. 2024-03-26T21:53:58.661 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 systemd[1]: Starting Ceph mon.smithi194 for 2f9b9224-ebbb-11ee-95d0-87774f69a715... 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 podman[29674]: 2024-03-26 21:53:58.66020589 +0000 UTC m=+0.018037245 image pull 4c6f72a25bfc8f895430ebfdfa5b9e4c4e63b2b11225a224873d069d9b227ce1 quay.ceph.io/ceph-ci/ceph:reef 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 podman[29674]: 2024-03-26 21:53:58.763023343 +0000 UTC m=+0.120854745 container create f276715a9ec198666af36d2136c0ec3b32b5e44ff8fd349b66d1c6b877280239 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-d83b58f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 podman[29674]: 2024-03-26 21:53:58.940082004 +0000 UTC m=+0.297913360 container init f276715a9ec198666af36d2136c0ec3b32b5e44ff8fd349b66d1c6b877280239 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=reef-d83b58f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 podman[29674]: 2024-03-26 21:53:58.942928195 +0000 UTC m=+0.300759555 container start f276715a9ec198666af36d2136c0ec3b32b5e44ff8fd349b66d1c6b877280239 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=reef-d83b58f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: ceph version 18.2.2-983-gd83b58fa (d83b58fabace7c0b0c2d45a3ac2899524ef89e97) reef (stable), process ceph-mon, pid 2 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: pidfile_write: ignore empty --pid-file 2024-03-26T21:53:59.004 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: load: jerasure load: lrc 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: RocksDB version: 7.9.2 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Git sha 0 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Compile date 2024-03-25 22:27:05 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: DB SUMMARY 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: DB Session ID: 530TRPKGO6W1E1LAQCRL 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: CURRENT file: CURRENT 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi194/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi194/store.db: 000009.log size: 98986 ; 2024-03-26T21:53:59.005 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.error_if_exists: 0 2024-03-26T21:53:59.006 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.create_if_missing: 0 2024-03-26T21:53:59.006 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.paranoid_checks: 1 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.env: 0x55c797c17720 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.info_log: 0x55c79a097380 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.statistics: (nil) 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.use_fsync: 0 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_log_file_size: 0 2024-03-26T21:53:59.007 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.allow_fallocate: 1 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.use_direct_reads: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.db_log_dir: 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.wal_dir: 2024-03-26T21:53:59.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.write_buffer_manager: 0x55c79930e140 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T21:53:59.009 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.unordered_write: 0 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.row_cache: None 2024-03-26T21:53:59.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.wal_filter: None 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.wal_compression: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:53:59.011 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:53:59.012 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_open_files: -1 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Compression algorithms supported: 2024-03-26T21:53:59.013 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kZSTD supported: 0 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:53:59.014 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi194/store.db/MANIFEST-000010 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.merge_operator: 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_filter: None 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:53:59.015 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c79a0974a0) 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: index_type: 0 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: data_block_index_type: 0 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: index_shortening: 1 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:53:59.016 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: checksum: 4 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: no_block_cache: 0 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_cache: 0x55c7993af1f0 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_cache_options: 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: capacity : 536870912 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: num_shard_bits : 4 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: strict_capacity_limit : 0 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_cache_compressed: (nil) 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: persistent_cache: (nil) 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_size: 4096 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_size_deviation: 10 2024-03-26T21:53:59.017 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_restart_interval: 16 2024-03-26T21:53:59.018 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: index_block_restart_interval: 1 2024-03-26T21:53:59.018 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: metadata_block_size: 4096 2024-03-26T21:53:59.018 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: partition_filters: 0 2024-03-26T21:53:59.018 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: use_delta_encoding: 1 2024-03-26T21:53:59.018 INFO:teuthology.orchestra.run.smithi194.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T21:53:59.019 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: filter_policy: bloomfilter 2024-03-26T21:53:59.019 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: whole_key_filtering: 1 2024-03-26T21:53:59.019 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: verify_compression: 0 2024-03-26T21:53:59.019 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:53:59.019 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: format_version: 5 2024-03-26T21:53:59.019 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: enable_index_compression: 1 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: block_align: 0 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: max_auto_readahead_size: 262144 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: prepopulate_block_cache: 0 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression: NoCompression 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:53:59.020 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.num_levels: 7 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:53:59.021 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:53:59.022 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:53:59.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:53:59.024 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.table_properties_collectors: 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:53:59.025 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.ttl: 2592000 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:53:59.026 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi194/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 874998c0-9525-4802-8066-ac16bbfa0a2f 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711490038971708, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T21:53:59.027 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711490038972700, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94019, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 323, "table_properties": {"data_size": 92053, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13592, "raw_average_key_size": 49, "raw_value_size": 84604, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 274, "num_filter_entries": 274, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711490038, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "874998c0-9525-4802-8066-ac16bbfa0a2f", "db_session_id": "530TRPKGO6W1E1LAQCRL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711490038972782, "job": 1, "event": "recovery_finished"} 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi194/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c79944c000 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: DB pointer 0x55c79943a000 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: ** DB Stats ** 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:53:59.028 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: ** Compaction Stats [default] ** 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: L0 2/0 93.69 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Sum 2/0 93.69 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: ** Compaction Stats [default] ** 2024-03-26T21:53:59.029 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:53:59.030 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Cumulative compaction: 0.00 GB write, 30.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Interval compaction: 0.00 GB write, 30.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Block cache BinnedLRUCache@0x55c7993af1f0#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: starting mon.smithi194 rank 0 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi194 fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???) e1 preinit fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).mds e1 new map 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).mds e1 print_map 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: e1 2024-03-26T21:53:59.031 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: legacy client fscid: -1 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout: No filesystems configured 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:58 smithi194 ceph-mon[29693]: mon.smithi194@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 bash[29674]: f276715a9ec198666af36d2136c0ec3b32b5e44ff8fd349b66d1c6b877280239 2024-03-26T21:53:59.032 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 systemd[1]: Started Ceph mon.smithi194 for 2f9b9224-ebbb-11ee-95d0-87774f69a715. 2024-03-26T21:53:59.312 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 ceph-mon[29693]: mon.smithi194 is new leader, mons smithi194 in quorum (ranks 0) 2024-03-26T21:53:59.312 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 ceph-mon[29693]: monmap e1: 1 mons at {smithi194=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T21:53:59.313 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 ceph-mon[29693]: fsmap 2024-03-26T21:53:59.313 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 ceph-mon[29693]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T21:53:59.313 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:53:59 smithi194 ceph-mon[29693]: mgrmap e1: no daemons active 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.422+0000 7f1273b3a700 1 Processor -- start 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.423+0000 7f1273b3a700 1 -- start start 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.423+0000 7f1273b3a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.423+0000 7f1273b3a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c079e90 con 0x7f126c106f30 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.423+0000 7f12718d6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.423+0000 7f12718d6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:43806/0 (socket says 172.21.15.194:43806) 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.423+0000 7f12718d6700 1 -- 172.21.15.194:0/2748881017 learned_addr learned my addr 172.21.15.194:0/2748881017 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f12718d6700 1 -- 172.21.15.194:0/2748881017 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c109860 con 0x7f126c106f30 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f12718d6700 1 --2- 172.21.15.194:0/2748881017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c109320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f12680098a0 tx=0x7f1268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49df294b0ca4a30d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:00.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f12708d4700 1 -- 172.21.15.194:0/2748881017 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1268004180 con 0x7f126c106f30 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f12708d4700 1 -- 172.21.15.194:0/2748881017 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f12680042e0 con 0x7f126c106f30 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f12708d4700 1 -- 172.21.15.194:0/2748881017 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1268011a70 con 0x7f126c106f30 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f1273b3a700 1 -- 172.21.15.194:0/2748881017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 msgr2=0x7f126c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.424+0000 7f1273b3a700 1 --2- 172.21.15.194:0/2748881017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c109320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f12680098a0 tx=0x7f1268009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 -- 172.21.15.194:0/2748881017 shutdown_connections 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 --2- 172.21.15.194:0/2748881017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c109320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 -- 172.21.15.194:0/2748881017 >> 172.21.15.194:0/2748881017 conn(0x7f126c100980 msgr2=0x7f126c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 -- 172.21.15.194:0/2748881017 shutdown_connections 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 -- 172.21.15.194:0/2748881017 wait complete. 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 Processor -- start 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 -- start start 2024-03-26T21:54:00.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.425+0000 7f1273b3a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c19a710 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f12718d6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f12718d6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:43816/0 (socket says 172.21.15.194:43816) 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f12718d6700 1 -- 172.21.15.194:0/1162657453 learned_addr learned my addr 172.21.15.194:0/1162657453 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f12718d6700 1 -- 172.21.15.194:0/1162657453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268009550 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f12718d6700 1 --2- 172.21.15.194:0/1162657453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c19a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1268011a20 tx=0x7f12680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1268003b10 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c19a910 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f126801b8c0 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12680237b0 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c19adb0 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1268023d30 con 0x7f126c106f30 2024-03-26T21:54:00.034 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.426+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1268019710 con 0x7f126c106f30 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.427+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c0623e0 con 0x7f126c106f30 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.430+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f126801ba30 con 0x7f126c106f30 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.496+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f126c10a5e0 con 0x7f126c106f30 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.501+0000 7f1262ffd700 1 -- 172.21.15.194:0/1162657453 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f126801bbd0 con 0x7f126c106f30 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 msgr2=0x7f126c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 --2- 172.21.15.194:0/1162657453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c19a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1268011a20 tx=0x7f12680047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 shutdown_connections 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 --2- 172.21.15.194:0/1162657453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c106f30 0x7f126c19a1d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 >> 172.21.15.194:0/1162657453 conn(0x7f126c100980 msgr2=0x7f126c101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 shutdown_connections 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:53:59.502+0000 7f1273b3a700 1 -- 172.21.15.194:0/1162657453 wait complete. 2024-03-26T21:54:00.035 INFO:teuthology.orchestra.run.smithi194.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T21:54:00.057 INFO:teuthology.orchestra.run.smithi194.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:54:00.057 INFO:teuthology.orchestra.run.smithi194.stdout:Creating mgr... 2024-03-26T21:54:00.057 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T21:54:00.058 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T21:54:00.058 INFO:teuthology.orchestra.run.smithi194.stdout:Verifying port 0.0.0.0:8443 ... 2024-03-26T21:54:00.306 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mgr.smithi194.wndbij 2024-03-26T21:54:00.306 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mgr.smithi194.wndbij.service: Unit ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mgr.smithi194.wndbij.service not loaded. 2024-03-26T21:54:00.449 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715.target.wants/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mgr.smithi194.wndbij.service → /etc/systemd/system/ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@.service. 2024-03-26T21:54:00.765 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:00 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1162657453' entity='client.admin' 2024-03-26T21:54:00.975 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:54:00.975 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout disabled 2024-03-26T21:54:00.982 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:54:00.982 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout inactive 2024-03-26T21:54:00.982 INFO:teuthology.orchestra.run.smithi194.stdout:firewalld.service is not enabled 2024-03-26T21:54:00.982 INFO:teuthology.orchestra.run.smithi194.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T21:54:00.990 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:54:00.990 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout disabled 2024-03-26T21:54:00.997 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:54:00.997 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout inactive 2024-03-26T21:54:00.998 INFO:teuthology.orchestra.run.smithi194.stdout:firewalld.service is not enabled 2024-03-26T21:54:00.998 INFO:teuthology.orchestra.run.smithi194.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-03-26T21:54:00.998 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for mgr to start... 2024-03-26T21:54:00.998 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for mgr... 2024-03-26T21:54:01.660 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:01 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1679914451' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:54:02.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:54:02.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:02.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "fsid": "2f9b9224-ebbb-11ee-95d0-87774f69a715", 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:54:02.032 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "smithi194" 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:54:02.033 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:54:02.041 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.042 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:54:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:53:48.833097+0000", 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.412+0000 7f0c7c91e700 1 Processor -- start 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.412+0000 7f0c7c91e700 1 -- start start 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.412+0000 7f0c7c91e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.412+0000 7f0c7c91e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780745f0 con 0x7f0c78106f30 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.412+0000 7f0c7659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:43842/0 (socket says 172.21.15.194:43842) 2024-03-26T21:54:02.044 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7659c700 1 -- 172.21.15.194:0/2923997673 learned_addr learned my addr 172.21.15.194:0/2923997673 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7659c700 1 -- 172.21.15.194:0/2923997673 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c78109860 con 0x7f0c78106f30 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7659c700 1 --2- 172.21.15.194:0/2923997673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78109320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0c680098a0 tx=0x7f0c68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e446ad3c28f3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7559a700 1 -- 172.21.15.194:0/2923997673 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c68004180 con 0x7f0c78106f30 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7559a700 1 -- 172.21.15.194:0/2923997673 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0c680042e0 con 0x7f0c78106f30 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.413+0000 7f0c7559a700 1 -- 172.21.15.194:0/2923997673 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c68011a70 con 0x7f0c78106f30 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 -- 172.21.15.194:0/2923997673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 msgr2=0x7f0c78109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 --2- 172.21.15.194:0/2923997673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78109320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0c680098a0 tx=0x7f0c68009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 -- 172.21.15.194:0/2923997673 shutdown_connections 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 --2- 172.21.15.194:0/2923997673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78109320 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 -- 172.21.15.194:0/2923997673 >> 172.21.15.194:0/2923997673 conn(0x7f0c781009c0 msgr2=0x7f0c78102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 -- 172.21.15.194:0/2923997673 shutdown_connections 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.414+0000 7f0c7c91e700 1 -- 172.21.15.194:0/2923997673 wait complete. 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7c91e700 1 Processor -- start 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7c91e700 1 -- start start 2024-03-26T21:54:02.045 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7c91e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7c91e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78106ca0 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:43856/0 (socket says 172.21.15.194:43856) 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7659c700 1 -- 172.21.15.194:0/1679914451 learned_addr learned my addr 172.21.15.194:0/1679914451 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7659c700 1 -- 172.21.15.194:0/1679914451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68009550 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.415+0000 7f0c7659c700 1 --2- 172.21.15.194:0/1679914451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78106760 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0c68011710 tx=0x7f0c680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.416+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c68003b10 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.416+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0c6801b8c0 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.416+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c78104ee0 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.416+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c78105380 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.417+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c680197e0 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.417+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0c68019d40 con 0x7f0c78106f30 2024-03-26T21:54:02.046 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.417+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0c6801a5b0 con 0x7f0c78106f30 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.417+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7810a5e0 con 0x7f0c78106f30 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.421+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f0c68029060 con 0x7f0c78106f30 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.494+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0c7810a7d0 con 0x7f0c78106f30 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.494+0000 7f0c677fe700 1 -- 172.21.15.194:0/1679914451 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f0c6801f070 con 0x7f0c78106f30 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 msgr2=0x7f0c78106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 --2- 172.21.15.194:0/1679914451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78106760 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0c68011710 tx=0x7f0c680047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 shutdown_connections 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 --2- 172.21.15.194:0/1679914451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0c78106f30 0x7f0c78106760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 >> 172.21.15.194:0/1679914451 conn(0x7f0c781009c0 msgr2=0x7f0c781014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 shutdown_connections 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:01.495+0000 7f0c7c91e700 1 -- 172.21.15.194:0/1679914451 wait complete. 2024-03-26T21:54:02.047 INFO:teuthology.orchestra.run.smithi194.stdout:mgr not available, waiting (1/15)... 2024-03-26T21:54:06.410 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:06 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1890853580' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "fsid": "2f9b9224-ebbb-11ee-95d0-87774f69a715", 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:54:06.504 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "smithi194" 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:54:06.505 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:54:06.506 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:53:48.833097+0000", 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.850+0000 7fbe536ed700 1 Processor -- start 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.850+0000 7fbe536ed700 1 -- start start 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.850+0000 7fbe536ed700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c106940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.850+0000 7fbe536ed700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c106f10 con 0x7fbe4c106520 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe51489700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe51489700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:45632/0 (socket says 172.21.15.194:45632) 2024-03-26T21:54:06.508 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe51489700 1 -- 172.21.15.194:0/3360337910 learned_addr learned my addr 172.21.15.194:0/3360337910 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe51489700 1 -- 172.21.15.194:0/3360337910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c107680 con 0x7fbe4c106520 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe51489700 1 --2- 172.21.15.194:0/3360337910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c106940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0098a0 tx=0x7fbe3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6587c570534912c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe43fff700 1 -- 172.21.15.194:0/3360337910 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3c004180 con 0x7fbe4c106520 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe43fff700 1 -- 172.21.15.194:0/3360337910 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbe3c0042e0 con 0x7fbe4c106520 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.851+0000 7fbe43fff700 1 -- 172.21.15.194:0/3360337910 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3c011a70 con 0x7fbe4c106520 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 -- 172.21.15.194:0/3360337910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 msgr2=0x7fbe4c106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 --2- 172.21.15.194:0/3360337910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c106940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0098a0 tx=0x7fbe3c009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 -- 172.21.15.194:0/3360337910 shutdown_connections 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 --2- 172.21.15.194:0/3360337910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c106940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:06.509 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 -- 172.21.15.194:0/3360337910 >> 172.21.15.194:0/3360337910 conn(0x7fbe4c101b40 msgr2=0x7fbe4c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 -- 172.21.15.194:0/3360337910 shutdown_connections 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.852+0000 7fbe536ed700 1 -- 172.21.15.194:0/3360337910 wait complete. 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe536ed700 1 Processor -- start 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe536ed700 1 -- start start 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe536ed700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe536ed700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c19a6f0 con 0x7fbe4c106520 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe51489700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe51489700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:45648/0 (socket says 172.21.15.194:45648) 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe51489700 1 -- 172.21.15.194:0/1890853580 learned_addr learned my addr 172.21.15.194:0/1890853580 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe51489700 1 -- 172.21.15.194:0/1890853580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c009550 con 0x7fbe4c106520 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.853+0000 7fbe51489700 1 --2- 172.21.15.194:0/1890853580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c19a1b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c011710 tx=0x7fbe3c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3c003b10 con 0x7fbe4c106520 2024-03-26T21:54:06.510 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c19a8f0 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbe3c01b8c0 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3c019770 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c19ad90 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbe3c019cf0 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.854+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbe3c01a710 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.855+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c0623e0 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.859+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fbe3c029060 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.958+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbe4c10a8b0 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.958+0000 7fbe427fc700 1 -- 172.21.15.194:0/1890853580 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fbe3c01f070 con 0x7fbe4c106520 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 msgr2=0x7fbe4c19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 --2- 172.21.15.194:0/1890853580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c19a1b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c011710 tx=0x7fbe3c0047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 shutdown_connections 2024-03-26T21:54:06.512 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 --2- 172.21.15.194:0/1890853580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbe4c106520 0x7fbe4c19a1b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:06.512 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 >> 172.21.15.194:0/1890853580 conn(0x7fbe4c101b40 msgr2=0x7fbe4c102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:06.512 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 shutdown_connections 2024-03-26T21:54:06.512 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:05.960+0000 7fbe536ed700 1 -- 172.21.15.194:0/1890853580 wait complete. 2024-03-26T21:54:06.512 INFO:teuthology.orchestra.run.smithi194.stdout:mgr not available, waiting (2/15)... 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: Activating manager daemon smithi194.wndbij 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: mgrmap e2: smithi194.wndbij(active, starting, since 0.00295771s) 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mon metadata", "id": "smithi194"}]: dispatch 2024-03-26T21:54:08.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mgr metadata", "who": "smithi194.wndbij", "id": "smithi194.wndbij"}]: dispatch 2024-03-26T21:54:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: Manager daemon smithi194.wndbij is now available 2024-03-26T21:54:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi194.wndbij/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:54:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi194.wndbij/trash_purge_schedule"}]: dispatch 2024-03-26T21:54:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:07 smithi194 ceph-mon[29693]: from='mgr.14100 172.21.15.194:0/3646552620' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "fsid": "2f9b9224-ebbb-11ee-95d0-87774f69a715", 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:54:09.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "smithi194" 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:54:09.746 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:54:09.747 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:54:09.754 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:54:09.754 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.754 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:54:09.754 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:54:09.754 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:54:09.754 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:54:09.755 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:53:48.833097+0000", 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.925+0000 7f9518533700 1 Processor -- start 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.925+0000 7f9518533700 1 -- start start 2024-03-26T21:54:09.756 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f9518533700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f9510106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f9518533700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510106f00 con 0x7f9510106510 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f95162cf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f9510106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f95162cf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f9510106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:45700/0 (socket says 172.21.15.194:45700) 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f95162cf700 1 -- 172.21.15.194:0/2687656468 learned_addr learned my addr 172.21.15.194:0/2687656468 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f95162cf700 1 -- 172.21.15.194:0/2687656468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95101076c0 con 0x7f9510106510 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.926+0000 7f95162cf700 1 --2- 172.21.15.194:0/2687656468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f9510106930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f95080098a0 tx=0x7f9508009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7dc0830bdcf418b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f95152cd700 1 -- 172.21.15.194:0/2687656468 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9508004180 con 0x7f9510106510 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f95152cd700 1 -- 172.21.15.194:0/2687656468 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f95080042e0 con 0x7f9510106510 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f95152cd700 1 -- 172.21.15.194:0/2687656468 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9508011a70 con 0x7f9510106510 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 -- 172.21.15.194:0/2687656468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 msgr2=0x7f9510106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 --2- 172.21.15.194:0/2687656468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f9510106930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f95080098a0 tx=0x7f9508009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 -- 172.21.15.194:0/2687656468 shutdown_connections 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 --2- 172.21.15.194:0/2687656468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f9510106930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:09.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 -- 172.21.15.194:0/2687656468 >> 172.21.15.194:0/2687656468 conn(0x7f9510101b10 msgr2=0x7f9510103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 -- 172.21.15.194:0/2687656468 shutdown_connections 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.927+0000 7f9518533700 1 -- 172.21.15.194:0/2687656468 wait complete. 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f9518533700 1 Processor -- start 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f9518533700 1 -- start start 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f9518533700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f951019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f9518533700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951019a790 con 0x7f9510106510 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f95162cf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f951019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f95162cf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f951019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:45714/0 (socket says 172.21.15.194:45714) 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f95162cf700 1 -- 172.21.15.194:0/2975212204 learned_addr learned my addr 172.21.15.194:0/2975212204 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f95162cf700 1 -- 172.21.15.194:0/2975212204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508009550 con 0x7f9510106510 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.928+0000 7f95162cf700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f951019a250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9508011710 tx=0x7f95080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.929+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9508003b10 con 0x7f9510106510 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.929+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f950801b8c0 con 0x7f9510106510 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.929+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951019a990 con 0x7f9510106510 2024-03-26T21:54:09.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.929+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9508019770 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.929+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951019ae30 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.929+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 3) v1 ==== 45431+0+0 (secure 0 0 0) 0x7f95080199f0 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.930+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951010a8f0 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.930+0000 7f95077fe700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f94fc0385b0 0x7f94fc03aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.930+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f950804c980 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.930+0000 7f9515ace700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f94fc0385b0 0x7f94fc03aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.930+0000 7f9515ace700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f94fc0385b0 0x7f94fc03aa70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9500006fd0 tx=0x7f9500006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:08.935+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f950801a750 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.207+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f951019d6d0 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.208+0000 7f95077fe700 1 -- 172.21.15.194:0/2975212204 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9508018350 con 0x7f9510106510 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f94fc0385b0 msgr2=0x7f94fc03aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f94fc0385b0 0x7f94fc03aa70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9500006fd0 tx=0x7f9500006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 msgr2=0x7f951019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f951019a250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9508011710 tx=0x7f95080047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:09.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 shutdown_connections 2024-03-26T21:54:09.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f94fc0385b0 0x7f94fc03aa70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:09.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 --2- 172.21.15.194:0/2975212204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9510106510 0x7f951019a250 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:09.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 >> 172.21.15.194:0/2975212204 conn(0x7f9510101b10 msgr2=0x7f95101026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:09.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 shutdown_connections 2024-03-26T21:54:09.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:09.211+0000 7f9518533700 1 -- 172.21.15.194:0/2975212204 wait complete. 2024-03-26T21:54:09.760 INFO:teuthology.orchestra.run.smithi194.stdout:mgr is available 2024-03-26T21:54:10.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:09 smithi194 ceph-mon[29693]: mgrmap e3: smithi194.wndbij(active, since 1.00721s) 2024-03-26T21:54:10.008 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:09 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/2975212204' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:54:11.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:10 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/2024013505' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T21:54:12.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:11 smithi194 ceph-mon[29693]: mgrmap e4: smithi194.wndbij(active, since 3s) 2024-03-26T21:54:12.433 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout fsid = 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:54:12.440 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.112+0000 7f13808d5700 1 Processor -- start 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.112+0000 7f13808d5700 1 -- start start 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.112+0000 7f13808d5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.112+0000 7f13808d5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c07a8b0 con 0x7f137c07be30 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:12.441 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:45728/0 (socket says 172.21.15.194:45728) 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137a59c700 1 -- 172.21.15.194:0/3724937703 learned_addr learned my addr 172.21.15.194:0/3724937703 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137a59c700 1 -- 172.21.15.194:0/3724937703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c07a9f0 con 0x7f137c07be30 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137a59c700 1 --2- 172.21.15.194:0/3724937703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c07a2e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f136c0098a0 tx=0x7f136c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c212be5ab0d6ad38 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137959a700 1 -- 172.21.15.194:0/3724937703 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c004180 con 0x7f137c07be30 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137959a700 1 -- 172.21.15.194:0/3724937703 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f136c0042e0 con 0x7f137c07be30 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.113+0000 7f137959a700 1 -- 172.21.15.194:0/3724937703 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c011a70 con 0x7f137c07be30 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 -- 172.21.15.194:0/3724937703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 msgr2=0x7f137c07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 --2- 172.21.15.194:0/3724937703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c07a2e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f136c0098a0 tx=0x7f136c009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 -- 172.21.15.194:0/3724937703 shutdown_connections 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 --2- 172.21.15.194:0/3724937703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c07a2e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 -- 172.21.15.194:0/3724937703 >> 172.21.15.194:0/3724937703 conn(0x7f137c101b40 msgr2=0x7f137c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 -- 172.21.15.194:0/3724937703 shutdown_connections 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.114+0000 7f13808d5700 1 -- 172.21.15.194:0/3724937703 wait complete. 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f13808d5700 1 Processor -- start 2024-03-26T21:54:12.442 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f13808d5700 1 -- start start 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f13808d5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f13808d5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c19c8c0 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f137a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f137a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c19c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:45732/0 (socket says 172.21.15.194:45732) 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f137a59c700 1 -- 172.21.15.194:0/2024013505 learned_addr learned my addr 172.21.15.194:0/2024013505 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f137a59c700 1 -- 172.21.15.194:0/2024013505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c009550 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.115+0000 7f137a59c700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c19c380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f136c011710 tx=0x7f136c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.116+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c003b10 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.116+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f136c01b8c0 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.116+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c19cac0 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.116+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c0197e0 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.116+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c19cf60 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.116+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 3) v1 ==== 45431+0+0 (secure 0 0 0) 0x7f136c0199d0 con 0x7f137c07be30 2024-03-26T21:54:12.443 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.117+0000 7f136b7fe700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f1364038600 0x7f136403aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.117+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f136c04c6a0 con 0x7f137c07be30 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.117+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c10c7e0 con 0x7f137c07be30 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.117+0000 7f1379d9b700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f1364038600 0x7f136403aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.117+0000 7f1379d9b700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f1364038600 0x7f136403aac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1370006fd0 tx=0x7f1370006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.122+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f136c01f080 con 0x7f137c07be30 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.319+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f137c19f7f0 con 0x7f137c07be30 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.320+0000 7f136b7fe700 1 -- 172.21.15.194:0/2024013505 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f136c047030 con 0x7f137c07be30 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f1364038600 msgr2=0x7f136403aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f1364038600 0x7f136403aac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1370006fd0 tx=0x7f1370006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 msgr2=0x7f137c19c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c19c380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f136c011710 tx=0x7f136c0047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 shutdown_connections 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f1364038600 0x7f136403aac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 --2- 172.21.15.194:0/2024013505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f137c07be30 0x7f137c19c380 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:12.444 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 >> 172.21.15.194:0/2024013505 conn(0x7f137c101b40 msgr2=0x7f137c102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:12.445 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 shutdown_connections 2024-03-26T21:54:12.445 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:10.322+0000 7f13808d5700 1 -- 172.21.15.194:0/2024013505 wait complete. 2024-03-26T21:54:12.445 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling cephadm module... 2024-03-26T21:54:13.410 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:13 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/3252495564' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.835+0000 7f906c7cb700 1 Processor -- start 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.835+0000 7f906c7cb700 1 -- start start 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.835+0000 7f906c7cb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f9064106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.835+0000 7f906c7cb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9064106f40 con 0x7f9064106550 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f906a567700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f9064106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f906a567700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f9064106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:59968/0 (socket says 172.21.15.194:59968) 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f906a567700 1 -- 172.21.15.194:0/1232253042 learned_addr learned my addr 172.21.15.194:0/1232253042 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f906a567700 1 -- 172.21.15.194:0/1232253042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90641076b0 con 0x7f9064106550 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f906a567700 1 --2- 172.21.15.194:0/1232253042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f9064106970 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9060009e80 tx=0x7f906000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a132a5ff83b14cd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f9069565700 1 -- 172.21.15.194:0/1232253042 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f906000e070 con 0x7f9064106550 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f9069565700 1 -- 172.21.15.194:0/1232253042 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f906000b8c0 con 0x7f9064106550 2024-03-26T21:54:13.686 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.836+0000 7f9069565700 1 -- 172.21.15.194:0/1232253042 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9060004440 con 0x7f9064106550 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 -- 172.21.15.194:0/1232253042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 msgr2=0x7f9064106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 --2- 172.21.15.194:0/1232253042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f9064106970 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9060009e80 tx=0x7f906000b330 comp rx=0 tx=0).stop 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 -- 172.21.15.194:0/1232253042 shutdown_connections 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 --2- 172.21.15.194:0/1232253042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f9064106970 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 -- 172.21.15.194:0/1232253042 >> 172.21.15.194:0/1232253042 conn(0x7f9064101b10 msgr2=0x7f9064103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 -- 172.21.15.194:0/1232253042 shutdown_connections 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 -- 172.21.15.194:0/1232253042 wait complete. 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 Processor -- start 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.837+0000 7f906c7cb700 1 -- start start 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906c7cb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f906419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906c7cb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906419a6e0 con 0x7f9064106550 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906a567700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f906419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906a567700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f906419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:59974/0 (socket says 172.21.15.194:59974) 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906a567700 1 -- 172.21.15.194:0/3252495564 learned_addr learned my addr 172.21.15.194:0/3252495564 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906a567700 1 -- 172.21.15.194:0/3252495564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060009b30 con 0x7f9064106550 2024-03-26T21:54:13.687 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906a567700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f906419a1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f90600099e0 tx=0x7f9060003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f906000e040 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f906000bc40 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906419a8e0 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90600125e0 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.838+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906419ade0 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.839+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f9060019040 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.839+0000 7f905b7fe700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f90500386b0 0x7f905003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.839+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f906004bd00 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.839+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906410a8e0 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.839+0000 7f9069d66700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f90500386b0 0x7f905003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.840+0000 7f9069d66700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f90500386b0 0x7f905003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9054006fd0 tx=0x7f9054006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:12.844+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f906001f080 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.089+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f906419d610 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.142+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f906001ba40 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.143+0000 7f905b7fe700 1 -- 172.21.15.194:0/3252495564 <== mon.0 v2:172.21.15.194:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f906004ba60 con 0x7f9064106550 2024-03-26T21:54:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f90500386b0 msgr2=0x7f905003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f90500386b0 0x7f905003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9054006fd0 tx=0x7f9054006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 msgr2=0x7f906419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f906419a1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f90600099e0 tx=0x7f9060003d00 comp rx=0 tx=0).stop 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 shutdown_connections 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f90500386b0 0x7f905003ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 --2- 172.21.15.194:0/3252495564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064106550 0x7f906419a1a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.145+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 >> 172.21.15.194:0/3252495564 conn(0x7f9064101b10 msgr2=0x7f9064102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.146+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 shutdown_connections 2024-03-26T21:54:13.689 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:13.146+0000 7f906c7cb700 1 -- 172.21.15.194:0/3252495564 wait complete. 2024-03-26T21:54:14.345 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:14 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/3252495564' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T21:54:14.345 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:14 smithi194 ceph-mon[29693]: mgrmap e5: smithi194.wndbij(active, since 5s) 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "active_name": "smithi194.wndbij", 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.083+0000 7fdaf0720700 1 Processor -- start 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.083+0000 7fdaf0720700 1 -- start start 2024-03-26T21:54:14.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaf0720700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae81057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaf0720700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae8105db0 con 0x7fdae81053c0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaee4bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae81057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaee4bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae81057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:59992/0 (socket says 172.21.15.194:59992) 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaee4bc700 1 -- 172.21.15.194:0/2686632273 learned_addr learned my addr 172.21.15.194:0/2686632273 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaee4bc700 1 -- 172.21.15.194:0/2686632273 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae81065c0 con 0x7fdae81053c0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.084+0000 7fdaee4bc700 1 --2- 172.21.15.194:0/2686632273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae81057e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdad80098a0 tx=0x7fdad8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e16055f771fd9ce server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.085+0000 7fdaed4ba700 1 -- 172.21.15.194:0/2686632273 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdad8004180 con 0x7fdae81053c0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.085+0000 7fdaed4ba700 1 -- 172.21.15.194:0/2686632273 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdad80042e0 con 0x7fdae81053c0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.085+0000 7fdaed4ba700 1 -- 172.21.15.194:0/2686632273 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdad8011a70 con 0x7fdae81053c0 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.085+0000 7fdaf0720700 1 -- 172.21.15.194:0/2686632273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 msgr2=0x7fdae81057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.085+0000 7fdaf0720700 1 --2- 172.21.15.194:0/2686632273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae81057e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdad80098a0 tx=0x7fdad8009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 -- 172.21.15.194:0/2686632273 shutdown_connections 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 --2- 172.21.15.194:0/2686632273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae81057e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:14.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 -- 172.21.15.194:0/2686632273 >> 172.21.15.194:0/2686632273 conn(0x7fdae81009c0 msgr2=0x7fdae8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 -- 172.21.15.194:0/2686632273 shutdown_connections 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 -- 172.21.15.194:0/2686632273 wait complete. 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 Processor -- start 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.086+0000 7fdaf0720700 1 -- start start 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaf0720700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae8199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaf0720700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae8199650 con 0x7fdae81053c0 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaee4bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae8199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaee4bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae8199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:60004/0 (socket says 172.21.15.194:60004) 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaee4bc700 1 -- 172.21.15.194:0/4074888624 learned_addr learned my addr 172.21.15.194:0/4074888624 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaee4bc700 1 -- 172.21.15.194:0/4074888624 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad8009550 con 0x7fdae81053c0 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaee4bc700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae8199110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdad8011710 tx=0x7fdad80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdad8003b10 con 0x7fdae81053c0 2024-03-26T21:54:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdad801b8c0 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae8199850 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae8199cf0 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.087+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdad8019710 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.088+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fdad8019990 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.088+0000 7fdadf7fe700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 0x7fdad403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.088+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdad804dac0 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.088+0000 7fdaedcbb700 1 -- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 msgr2=0x7fdad403aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.088+0000 7fdaedcbb700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 0x7fdad403aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.088+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc005320 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.093+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdad8018d80 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.289+0000 7fdaedcbb700 1 -- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 msgr2=0x7fdad403aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.289+0000 7fdaedcbb700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 0x7fdad403aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.350+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fdacc005de0 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.350+0000 7fdadf7fe700 1 -- 172.21.15.194:0/4074888624 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fdad8018350 con 0x7fdae81053c0 2024-03-26T21:54:14.895 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 msgr2=0x7fdad403aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 0x7fdad403aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 msgr2=0x7fdae8199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae8199110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdad8011710 tx=0x7fdad80047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 shutdown_connections 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7fdad40386e0 0x7fdad403aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 --2- 172.21.15.194:0/4074888624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdae81053c0 0x7fdae8199110 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.352+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 >> 172.21.15.194:0/4074888624 conn(0x7fdae81009c0 msgr2=0x7fdae8101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.353+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 shutdown_connections 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:14.353+0000 7fdaf0720700 1 -- 172.21.15.194:0/4074888624 wait complete. 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for the mgr to restart... 2024-03-26T21:54:14.896 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for mgr epoch 5... 2024-03-26T21:54:15.320 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:15 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/4074888624' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T21:54:20.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: Active manager daemon smithi194.wndbij restarted 2024-03-26T21:54:20.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: Activating manager daemon smithi194.wndbij 2024-03-26T21:54:20.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T21:54:20.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: mgrmap e6: smithi194.wndbij(active, starting, since 0.00330617s) 2024-03-26T21:54:20.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mon metadata", "id": "smithi194"}]: dispatch 2024-03-26T21:54:20.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mgr metadata", "who": "smithi194.wndbij", "id": "smithi194.wndbij"}]: dispatch 2024-03-26T21:54:20.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:54:20.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:54:20.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:19 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: Manager daemon smithi194.wndbij is now available 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: Found migration_current of "None". Setting to last migration. 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi194.wndbij/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:54:21.264 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi194.wndbij/trash_purge_schedule"}]: dispatch 2024-03-26T21:54:21.265 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:21.265 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:21.265 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:20 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.333+0000 7f8b8db59700 1 Processor -- start 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.333+0000 7f8b8db59700 1 -- start start 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b8db59700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b881093e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b8db59700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b88079dd0 con 0x7f8b88106ff0 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b877fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b881093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b877fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b881093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:60006/0 (socket says 172.21.15.194:60006) 2024-03-26T21:54:21.533 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b877fe700 1 -- 172.21.15.194:0/2052260741 learned_addr learned my addr 172.21.15.194:0/2052260741 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b877fe700 1 -- 172.21.15.194:0/2052260741 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b88079f10 con 0x7f8b88106ff0 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b877fe700 1 --2- 172.21.15.194:0/2052260741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b881093e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c0098a0 tx=0x7f8b7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27687f7ddd72be7f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b867fc700 1 -- 172.21.15.194:0/2052260741 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b7c004180 con 0x7f8b88106ff0 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b867fc700 1 -- 172.21.15.194:0/2052260741 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8b7c0042e0 con 0x7f8b88106ff0 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.334+0000 7f8b867fc700 1 -- 172.21.15.194:0/2052260741 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b7c011a70 con 0x7f8b88106ff0 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 -- 172.21.15.194:0/2052260741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 msgr2=0x7f8b881093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 --2- 172.21.15.194:0/2052260741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b881093e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c0098a0 tx=0x7f8b7c009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 -- 172.21.15.194:0/2052260741 shutdown_connections 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 --2- 172.21.15.194:0/2052260741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b881093e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 -- 172.21.15.194:0/2052260741 >> 172.21.15.194:0/2052260741 conn(0x7f8b88100980 msgr2=0x7f8b88102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 -- 172.21.15.194:0/2052260741 shutdown_connections 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.335+0000 7f8b8db59700 1 -- 172.21.15.194:0/2052260741 wait complete. 2024-03-26T21:54:21.534 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b8db59700 1 Processor -- start 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b8db59700 1 -- start start 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b8db59700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b88106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b8db59700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b88106d60 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b877fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b88106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b877fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b88106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:60020/0 (socket says 172.21.15.194:60020) 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b877fe700 1 -- 172.21.15.194:0/1880056268 learned_addr learned my addr 172.21.15.194:0/1880056268 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.336+0000 7f8b877fe700 1 -- 172.21.15.194:0/1880056268 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c009550 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.337+0000 7f8b877fe700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b88106820 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c011a20 tx=0x7f8b7c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.337+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b7c003b10 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.337+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8b7c01b8c0 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.337+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b7c0237b0 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.337+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b88104fa0 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.337+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b88105440 con 0x7f8b88106ff0 2024-03-26T21:54:21.535 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.338+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f8b7c0239e0 con 0x7f8b88106ff0 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.338+0000 7f8b84ff9700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.338+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 --> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b7003b300 con 0x7f8b70038730 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.338+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8b7c04cae0 con 0x7f8b88106ff0 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.338+0000 7f8b86ffd700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 msgr2=0x7f8b7003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.338+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.539+0000 7f8b86ffd700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 msgr2=0x7f8b7003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.539+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.940+0000 7f8b86ffd700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 msgr2=0x7f8b7003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:15.940+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:16.741+0000 7f8b86ffd700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 msgr2=0x7f8b7003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:16.741+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:18.341+0000 7f8b86ffd700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 msgr2=0x7f8b7003abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/2696979115 2024-03-26T21:54:21.536 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:18.341+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:19.961+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mgrmap(e 6) v1 ==== 45315+0+0 (secure 0 0 0) 0x7f8b7c02c3f0 con 0x7f8b88106ff0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:19.961+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 msgr2=0x7f8b7003abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:19.961+0000 7f8b84ff9700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.985+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 7) v1 ==== 45442+0+0 (secure 0 0 0) 0x7f8b7c04e4b0 con 0x7f8b88106ff0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.985+0000 7f8b84ff9700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f8b7003bfc0 0x7f8b7003e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.985+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b7003b300 con 0x7f8b7003bfc0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.985+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f8b7003bfc0 0x7f8b7003e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.986+0000 7f8b86ffd700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f8b7003bfc0 0x7f8b7003e3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8b78003a10 tx=0x7f8b780092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.986+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f8b7003b300 con 0x7f8b7003bfc0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.992+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8b880623e0 con 0x7f8b7003bfc0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b84ff9700 1 -- 172.21.15.194:0/1880056268 <== mgr.14118 v2:172.21.15.194:6800/549918706 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8b880623e0 con 0x7f8b7003bfc0 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f8b7003bfc0 msgr2=0x7f8b7003e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f8b7003bfc0 0x7f8b7003e3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8b78003a10 tx=0x7f8b780092b0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.537 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 msgr2=0x7f8b88106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b88106820 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c011a20 tx=0x7f8b7c0047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 shutdown_connections 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f8b7003bfc0 0x7f8b7003e3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b88106ff0 0x7f8b88106820 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 --2- 172.21.15.194:0/1880056268 >> [v2:172.21.15.194:6800/2696979115,v1:172.21.15.194:6801/2696979115] conn(0x7f8b70038730 0x7f8b7003abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 >> 172.21.15.194:0/1880056268 conn(0x7f8b88100980 msgr2=0x7f8b881014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 shutdown_connections 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:20.993+0000 7f8b8db59700 1 -- 172.21.15.194:0/1880056268 wait complete. 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:mgr epoch 5 is available 2024-03-26T21:54:21.538 INFO:teuthology.orchestra.run.smithi194.stdout:Setting orchestrator backend to cephadm... 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:20] ENGINE Bus STARTING 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:20] ENGINE Serving on http://172.21.15.194:8765 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:20] ENGINE Serving on https://172.21.15.194:7150 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:20] ENGINE Bus STARTED 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: mgrmap e7: smithi194.wndbij(active, since 1.028s) 2024-03-26T21:54:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:21 smithi194 ceph-mon[29693]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T21:54:22.741 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f09152ba700 1 Processor -- start 2024-03-26T21:54:22.741 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f09152ba700 1 -- start start 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f09152ba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f09152ba700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091007a820 con 0x7f091007bde0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f090effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f090effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:60122/0 (socket says 172.21.15.194:60122) 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.953+0000 7f090effd700 1 -- 172.21.15.194:0/3368170446 learned_addr learned my addr 172.21.15.194:0/3368170446 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f090effd700 1 -- 172.21.15.194:0/3368170446 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091007a960 con 0x7f091007bde0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f090effd700 1 --2- 172.21.15.194:0/3368170446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091007a2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0900009e80 tx=0x7f090000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486d8c287e648e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f090dffb700 1 -- 172.21.15.194:0/3368170446 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f090000e070 con 0x7f091007bde0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f090dffb700 1 -- 172.21.15.194:0/3368170446 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f090000b8c0 con 0x7f091007bde0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f090dffb700 1 -- 172.21.15.194:0/3368170446 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0900004440 con 0x7f091007bde0 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f09152ba700 1 -- 172.21.15.194:0/3368170446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 msgr2=0x7f091007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f09152ba700 1 --2- 172.21.15.194:0/3368170446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091007a2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0900009e80 tx=0x7f090000b330 comp rx=0 tx=0).stop 2024-03-26T21:54:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f09152ba700 1 -- 172.21.15.194:0/3368170446 shutdown_connections 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f09152ba700 1 --2- 172.21.15.194:0/3368170446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091007a2e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f09152ba700 1 -- 172.21.15.194:0/3368170446 >> 172.21.15.194:0/3368170446 conn(0x7f0910101b10 msgr2=0x7f0910103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.954+0000 7f09152ba700 1 -- 172.21.15.194:0/3368170446 shutdown_connections 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.955+0000 7f09152ba700 1 -- 172.21.15.194:0/3368170446 wait complete. 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.955+0000 7f09152ba700 1 Processor -- start 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.955+0000 7f09152ba700 1 -- start start 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.955+0000 7f09152ba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.955+0000 7f09152ba700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091019c890 con 0x7f091007bde0 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f090effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f090effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091019c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:60132/0 (socket says 172.21.15.194:60132) 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f090effd700 1 -- 172.21.15.194:0/3701271675 learned_addr learned my addr 172.21.15.194:0/3701271675 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f090effd700 1 -- 172.21.15.194:0/3701271675 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0900009b30 con 0x7f091007bde0 2024-03-26T21:54:22.743 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f090effd700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091019c350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09000099e0 tx=0x7f0900003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f090000e040 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f090000bd50 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091019ca90 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09000125c0 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.956+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091019cf90 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.957+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 7) v1 ==== 45442+0+0 (secure 0 0 0) 0x7f0900019040 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.957+0000 7f08f7fff700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f08f80385c0 0x7f08f803aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.957+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091010c7f0 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.957+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f090004c1b0 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.957+0000 7f090e7fc700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f08f80385c0 0x7f08f803aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.958+0000 7f090e7fc700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f08f80385c0 0x7f08f803aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0904006fd0 tx=0x7f0904006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.962+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f090001f080 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:21.986+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f0900024400 con 0x7f091007bde0 2024-03-26T21:54:22.744 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.175+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f0910102ce0 con 0x7f08f80385c0 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.184+0000 7f08f7fff700 1 -- 172.21.15.194:0/3701271675 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0910102ce0 con 0x7f08f80385c0 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f08f80385c0 msgr2=0x7f08f803aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f08f80385c0 0x7f08f803aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0904006fd0 tx=0x7f0904006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 msgr2=0x7f091019c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091019c350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09000099e0 tx=0x7f0900003d00 comp rx=0 tx=0).stop 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 shutdown_connections 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f08f80385c0 0x7f08f803aa80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 --2- 172.21.15.194:0/3701271675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f091007bde0 0x7f091019c350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 >> 172.21.15.194:0/3701271675 conn(0x7f0910101b10 msgr2=0x7f09101025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 shutdown_connections 2024-03-26T21:54:22.745 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:22.186+0000 7f09152ba700 1 -- 172.21.15.194:0/3701271675 wait complete. 2024-03-26T21:54:23.334 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:22 smithi194 ceph-mon[29693]: mgrmap e8: smithi194.wndbij(active, since 2s) 2024-03-26T21:54:23.334 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:22 smithi194 ceph-mon[29693]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:23.334 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:22 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:23.334 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:22 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.135+0000 7fce8a2fd700 1 Processor -- start 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.135+0000 7fce8a2fd700 1 -- start start 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.135+0000 7fce8a2fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce841057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.135+0000 7fce8a2fd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84105db0 con 0x7fce841053c0 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.135+0000 7fce83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce841057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce841057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38286/0 (socket says 172.21.15.194:38286) 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce83fff700 1 -- 172.21.15.194:0/3200151082 learned_addr learned my addr 172.21.15.194:0/3200151082 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce83fff700 1 -- 172.21.15.194:0/3200151082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce841065c0 con 0x7fce841053c0 2024-03-26T21:54:24.027 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce83fff700 1 --2- 172.21.15.194:0/3200151082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce841057e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce740098a0 tx=0x7fce74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b21a8ddf125bdd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce82ffd700 1 -- 172.21.15.194:0/3200151082 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74004180 con 0x7fce841053c0 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce82ffd700 1 -- 172.21.15.194:0/3200151082 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fce740042e0 con 0x7fce841053c0 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.136+0000 7fce82ffd700 1 -- 172.21.15.194:0/3200151082 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74011a70 con 0x7fce841053c0 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 -- 172.21.15.194:0/3200151082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 msgr2=0x7fce841057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 --2- 172.21.15.194:0/3200151082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce841057e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce740098a0 tx=0x7fce74009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 -- 172.21.15.194:0/3200151082 shutdown_connections 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 --2- 172.21.15.194:0/3200151082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce841057e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 -- 172.21.15.194:0/3200151082 >> 172.21.15.194:0/3200151082 conn(0x7fce841009c0 msgr2=0x7fce84102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 -- 172.21.15.194:0/3200151082 shutdown_connections 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 -- 172.21.15.194:0/3200151082 wait complete. 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 Processor -- start 2024-03-26T21:54:24.028 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.137+0000 7fce8a2fd700 1 -- start start 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce8a2fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce84074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce8a2fd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84074680 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce84074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce84074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38302/0 (socket says 172.21.15.194:38302) 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce83fff700 1 -- 172.21.15.194:0/4216738212 learned_addr learned my addr 172.21.15.194:0/4216738212 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce83fff700 1 -- 172.21.15.194:0/4216738212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74009550 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce83fff700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce84074140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce74011710 tx=0x7fce740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74003b10 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fce7401b8c0 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8419cf60 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74019710 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.138+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8419d3a0 con 0x7fce841053c0 2024-03-26T21:54:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.139+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fce740198c0 con 0x7fce841053c0 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.139+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce841097f0 con 0x7fce841053c0 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.139+0000 7fce817fa700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fce6c038730 0x7fce6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.140+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce7404ca90 con 0x7fce841053c0 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.140+0000 7fce837fe700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fce6c038730 0x7fce6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.140+0000 7fce837fe700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fce6c038730 0x7fce6c03abf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce78006fd0 tx=0x7fce78006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.144+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fce74018b80 con 0x7fce841053c0 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.337+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fce84101c40 con 0x7fce6c038730 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.338+0000 7fce817fa700 1 -- 172.21.15.194:0/4216738212 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fce84101c40 con 0x7fce6c038730 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fce6c038730 msgr2=0x7fce6c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fce6c038730 0x7fce6c03abf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce78006fd0 tx=0x7fce78006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 msgr2=0x7fce84074140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:24.030 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce84074140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce74011710 tx=0x7fce740047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 shutdown_connections 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fce6c038730 0x7fce6c03abf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 --2- 172.21.15.194:0/4216738212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce841053c0 0x7fce84074140 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 >> 172.21.15.194:0/4216738212 conn(0x7fce841009c0 msgr2=0x7fce84101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 shutdown_connections 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:23.340+0000 7fce8a2fd700 1 -- 172.21.15.194:0/4216738212 wait complete. 2024-03-26T21:54:24.031 INFO:teuthology.orchestra.run.smithi194.stdout:Generating ssh key... 2024-03-26T21:54:25.108 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:24 smithi194 ceph-mon[29693]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:25.109 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:24 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:25.109 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:24 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:25.391 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f681d017700 1 Processor -- start 2024-03-26T21:54:25.391 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f681d017700 1 -- start start 2024-03-26T21:54:25.391 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f681d017700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f6818106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:25.391 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f681d017700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6818106f00 con 0x7f6818106510 2024-03-26T21:54:25.391 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f6816d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f6818106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:25.391 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f6816d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f6818106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38318/0 (socket says 172.21.15.194:38318) 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f6816d9d700 1 -- 172.21.15.194:0/2613833527 learned_addr learned my addr 172.21.15.194:0/2613833527 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.400+0000 7f6816d9d700 1 -- 172.21.15.194:0/2613833527 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68181076c0 con 0x7f6818106510 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f6816d9d700 1 --2- 172.21.15.194:0/2613833527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f6818106930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68080098a0 tx=0x7f6808009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7db05edf95beeb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f6815d9b700 1 -- 172.21.15.194:0/2613833527 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6808004180 con 0x7f6818106510 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f6815d9b700 1 -- 172.21.15.194:0/2613833527 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f68080042e0 con 0x7f6818106510 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f6815d9b700 1 -- 172.21.15.194:0/2613833527 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6808011a70 con 0x7f6818106510 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 -- 172.21.15.194:0/2613833527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 msgr2=0x7f6818106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 --2- 172.21.15.194:0/2613833527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f6818106930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68080098a0 tx=0x7f6808009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 -- 172.21.15.194:0/2613833527 shutdown_connections 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 --2- 172.21.15.194:0/2613833527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f6818106930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 -- 172.21.15.194:0/2613833527 >> 172.21.15.194:0/2613833527 conn(0x7f6818101b10 msgr2=0x7f6818103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 -- 172.21.15.194:0/2613833527 shutdown_connections 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.401+0000 7f681d017700 1 -- 172.21.15.194:0/2613833527 wait complete. 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.402+0000 7f681d017700 1 Processor -- start 2024-03-26T21:54:25.392 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.402+0000 7f681d017700 1 -- start start 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.402+0000 7f681d017700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f681819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.402+0000 7f681d017700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681819a790 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f6816d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f681819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f6816d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f681819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38320/0 (socket says 172.21.15.194:38320) 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f6816d9d700 1 -- 172.21.15.194:0/2289645180 learned_addr learned my addr 172.21.15.194:0/2289645180 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f6816d9d700 1 -- 172.21.15.194:0/2289645180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6808009550 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f6816d9d700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f681819a250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6808011710 tx=0x7f68080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6808003b10 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f680801b8c0 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681819a990 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6808019770 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.403+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681819ae30 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.404+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f68080199a0 con 0x7f6818106510 2024-03-26T21:54:25.393 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.404+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681810a8f0 con 0x7f6818106510 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.404+0000 7f67fffff700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f6800038770 0x7f680003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.404+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f680804c8c0 con 0x7f6818106510 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.404+0000 7f681659c700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f6800038770 0x7f680003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.405+0000 7f681659c700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f6800038770 0x7f680003ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f680c006fd0 tx=0x7f680c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.409+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6808018d90 con 0x7f6818106510 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.604+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f681819d6d0 con 0x7f6800038770 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.790+0000 7f67fffff700 1 -- 172.21.15.194:0/2289645180 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f681819d6d0 con 0x7f6800038770 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f6800038770 msgr2=0x7f680003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f6800038770 0x7f680003ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f680c006fd0 tx=0x7f680c006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 msgr2=0x7f681819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f681819a250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6808011710 tx=0x7f68080047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 shutdown_connections 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f6800038770 0x7f680003ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:25.394 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 --2- 172.21.15.194:0/2289645180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6818106510 0x7f681819a250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:25.395 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 >> 172.21.15.194:0/2289645180 conn(0x7f6818101b10 msgr2=0x7f68181026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:25.395 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 shutdown_connections 2024-03-26T21:54:25.395 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:24.792+0000 7f681d017700 1 -- 172.21.15.194:0/2289645180 wait complete. 2024-03-26T21:54:26.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:25 smithi194 ceph-mon[29693]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:26.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:25 smithi194 ceph-mon[29693]: Generating ssh key... 2024-03-26T21:54:26.449 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ppzNzLTpeplBS4fNfSgIWijhpJVnSscHATdj7ONvLCoKNYlNoGzDnJRht9erX01bWiOH4Z/bWqFOS2RAq1d50EE1N2NYCF3NA4i9iWR5J6Wnj8tdiRecO9/j/am+MUl0URPMiKUjqlSk8nxgD3CmJkouv7+em8Mv9eVVZwGQqiYXoNFVfwjNIYD1IGBhW0zzCoM0eZ+tdnUy7wUD1nWOXtPtB63tr4Yek9d9poVlVGmidAs6Y8JYxW9Bj3kaJgmHPEJwpTUXTBi3e7cR224JC17YeXV20PNAQ2oBMSuZDxBkYEE0hVVeVHagiZvu6op24VxoZYt/qQi1CDCE8kZQf2edYhwANDmRUFFhTtaYojjEihgQj1yBaG+pNmK8vv+A+1hn+oHszbKhEBaonliYPctqt/YS12YJj7cXv6v2g/8OzO54xBt0+0cQouyAHfZbbOckOgU1yt5fNGFyvF+JvAAB01d3Hiwtwk8sMBrSJPATkOcRaWdHBzS6fXyOCmM= ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:54:26.449 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ece512700 1 Processor -- start 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ece512700 1 -- start start 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ece512700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec8106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ece512700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec8106f00 con 0x7f0ec8106510 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ec7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec8106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ec7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec8106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38332/0 (socket says 172.21.15.194:38332) 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ec7fff700 1 -- 172.21.15.194:0/2812116786 learned_addr learned my addr 172.21.15.194:0/2812116786 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.765+0000 7f0ec7fff700 1 -- 172.21.15.194:0/2812116786 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec81076c0 con 0x7f0ec8106510 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ec7fff700 1 --2- 172.21.15.194:0/2812116786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec8106930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0eb80098a0 tx=0x7f0eb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fcbbc57b26aa85 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ec6ffd700 1 -- 172.21.15.194:0/2812116786 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8004180 con 0x7f0ec8106510 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ec6ffd700 1 -- 172.21.15.194:0/2812116786 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0eb80042e0 con 0x7f0ec8106510 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ec6ffd700 1 -- 172.21.15.194:0/2812116786 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8011a70 con 0x7f0ec8106510 2024-03-26T21:54:26.450 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 -- 172.21.15.194:0/2812116786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 msgr2=0x7f0ec8106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 --2- 172.21.15.194:0/2812116786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec8106930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0eb80098a0 tx=0x7f0eb8009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 -- 172.21.15.194:0/2812116786 shutdown_connections 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 --2- 172.21.15.194:0/2812116786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec8106930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 -- 172.21.15.194:0/2812116786 >> 172.21.15.194:0/2812116786 conn(0x7f0ec8101b10 msgr2=0x7f0ec8103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 -- 172.21.15.194:0/2812116786 shutdown_connections 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.766+0000 7f0ece512700 1 -- 172.21.15.194:0/2812116786 wait complete. 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ece512700 1 Processor -- start 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ece512700 1 -- start start 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ece512700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ece512700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec819a790 con 0x7f0ec8106510 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ec7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ec7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38342/0 (socket says 172.21.15.194:38342) 2024-03-26T21:54:26.451 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.767+0000 7f0ec7fff700 1 -- 172.21.15.194:0/205021065 learned_addr learned my addr 172.21.15.194:0/205021065 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ec7fff700 1 -- 172.21.15.194:0/205021065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb8009550 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ec7fff700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec819a250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8011710 tx=0x7f0eb80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8003b10 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0eb801b8c0 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec819a990 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8019770 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.768+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec819ae30 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.769+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f0eb80199f0 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.769+0000 7f0ec57fa700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f0eb0038720 0x7f0eb003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.769+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0eb804c940 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.769+0000 7f0ec77fe700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f0eb0038720 0x7f0eb003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.769+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec80623e0 con 0x7f0ec8106510 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.769+0000 7f0ec77fe700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f0eb0038720 0x7f0eb003abe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc006fd0 tx=0x7f0ebc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:26.452 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.774+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0eb801f020 con 0x7f0ec8106510 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.965+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ec8102db0 con 0x7f0eb0038720 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.966+0000 7f0ec57fa700 1 -- 172.21.15.194:0/205021065 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0ec8102db0 con 0x7f0eb0038720 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f0eb0038720 msgr2=0x7f0eb003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f0eb0038720 0x7f0eb003abe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc006fd0 tx=0x7f0ebc006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 msgr2=0x7f0ec819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec819a250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8011710 tx=0x7f0eb80047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 shutdown_connections 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f0eb0038720 0x7f0eb003abe0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 --2- 172.21.15.194:0/205021065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ec8106510 0x7f0ec819a250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.968+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 >> 172.21.15.194:0/205021065 conn(0x7f0ec8101b10 msgr2=0x7f0ec81026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.969+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 shutdown_connections 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:25.969+0000 7f0ece512700 1 -- 172.21.15.194:0/205021065 wait complete. 2024-03-26T21:54:26.453 INFO:teuthology.orchestra.run.smithi194.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:54:26.454 INFO:teuthology.orchestra.run.smithi194.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T21:54:26.454 INFO:teuthology.orchestra.run.smithi194.stdout:Adding host smithi194... 2024-03-26T21:54:27.059 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:26 smithi194 ceph-mon[29693]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:28.160 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:27 smithi194 ceph-mon[29693]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "addr": "172.21.15.194", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:29.234 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:28 smithi194 ceph-mon[29693]: Deploying cephadm binary to smithi194 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Added host 'smithi194' with addr '172.21.15.194' 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.854+0000 7ffa1627b700 1 Processor -- start 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.854+0000 7ffa1627b700 1 -- start start 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa1627b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10105d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa1627b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10106300 con 0x7ffa10105910 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa0ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa0ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38356/0 (socket says 172.21.15.194:38356) 2024-03-26T21:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa0ffff700 1 -- 172.21.15.194:0/511229748 learned_addr learned my addr 172.21.15.194:0/511229748 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa0ffff700 1 -- 172.21.15.194:0/511229748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10106630 con 0x7ffa10105910 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.855+0000 7ffa0ffff700 1 --2- 172.21.15.194:0/511229748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10105d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa000098a0 tx=0x7ffa00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b206490b2d3481fa server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa0effd700 1 -- 172.21.15.194:0/511229748 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa00004180 con 0x7ffa10105910 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa0effd700 1 -- 172.21.15.194:0/511229748 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ffa000042e0 con 0x7ffa10105910 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa0effd700 1 -- 172.21.15.194:0/511229748 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa00011a70 con 0x7ffa10105910 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 -- 172.21.15.194:0/511229748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 msgr2=0x7ffa10105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 --2- 172.21.15.194:0/511229748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10105d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa000098a0 tx=0x7ffa00009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 -- 172.21.15.194:0/511229748 shutdown_connections 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 --2- 172.21.15.194:0/511229748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10105d30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 -- 172.21.15.194:0/511229748 >> 172.21.15.194:0/511229748 conn(0x7ffa10100170 msgr2=0x7ffa10102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 -- 172.21.15.194:0/511229748 shutdown_connections 2024-03-26T21:54:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.856+0000 7ffa1627b700 1 -- 172.21.15.194:0/511229748 wait complete. 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa1627b700 1 Processor -- start 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa1627b700 1 -- start start 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa1627b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10190480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa1627b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa101909c0 con 0x7ffa10105910 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa0ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10190480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa0ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10190480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38366/0 (socket says 172.21.15.194:38366) 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa0ffff700 1 -- 172.21.15.194:0/1498128239 learned_addr learned my addr 172.21.15.194:0/1498128239 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.857+0000 7ffa0ffff700 1 -- 172.21.15.194:0/1498128239 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00009550 con 0x7ffa10105910 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.858+0000 7ffa0ffff700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10190480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa00011710 tx=0x7ffa000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.858+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa00003b10 con 0x7ffa10105910 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.858+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ffa0001b8c0 con 0x7ffa10105910 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.858+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1018ec00 con 0x7ffa10105910 2024-03-26T21:54:30.191 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.858+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa000197e0 con 0x7ffa10105910 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.858+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1018f0a0 con 0x7ffa10105910 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.859+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7ffa00019a20 con 0x7ffa10105910 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.859+0000 7ffa0d7fa700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff9f8038720 0x7ff9f803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.859+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1018fab0 con 0x7ffa10105910 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.859+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa0004cbd0 con 0x7ffa10105910 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.859+0000 7ffa0f7fe700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff9f8038720 0x7ff9f803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.859+0000 7ffa0f7fe700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff9f8038720 0x7ff9f803abe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa04006fd0 tx=0x7ffa04006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:26.864+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffa0001a3d0 con 0x7ffa10105910 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:27.063+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "addr": "172.21.15.194", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa10194010 con 0x7ff9f8038720 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.636+0000 7ffa0d7fa700 1 -- 172.21.15.194:0/1498128239 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ffa10194010 con 0x7ff9f8038720 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff9f8038720 msgr2=0x7ff9f803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff9f8038720 0x7ff9f803abe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa04006fd0 tx=0x7ffa04006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 msgr2=0x7ffa10190480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:30.192 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10190480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa00011710 tx=0x7ffa000047c0 comp rx=0 tx=0).stop 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 shutdown_connections 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff9f8038720 0x7ff9f803abe0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 --2- 172.21.15.194:0/1498128239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa10105910 0x7ffa10190480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 >> 172.21.15.194:0/1498128239 conn(0x7ffa10100170 msgr2=0x7ffa10101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 shutdown_connections 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:29.639+0000 7ffa1627b700 1 -- 172.21.15.194:0/1498128239 wait complete. 2024-03-26T21:54:30.193 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying mon service with default placement... 2024-03-26T21:54:30.795 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:30 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:30.795 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:30 smithi194 ceph-mon[29693]: Added host smithi194 2024-03-26T21:54:30.795 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:30 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:32.410 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T21:54:32.410 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.663+0000 7f86e15b8700 1 Processor -- start 2024-03-26T21:54:32.410 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.663+0000 7f86e15b8700 1 -- start start 2024-03-26T21:54:32.410 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.663+0000 7f86e15b8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc109370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:32.410 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.663+0000 7f86e15b8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc079e50 con 0x7f86dc106f80 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86daffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86daffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38376/0 (socket says 172.21.15.194:38376) 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86daffd700 1 -- 172.21.15.194:0/1839270407 learned_addr learned my addr 172.21.15.194:0/1839270407 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86daffd700 1 -- 172.21.15.194:0/1839270407 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc1098b0 con 0x7f86dc106f80 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86daffd700 1 --2- 172.21.15.194:0/1839270407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc109370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86d00098a0 tx=0x7f86d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45934d5afd2b3b45 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86d9ffb700 1 -- 172.21.15.194:0/1839270407 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0004180 con 0x7f86dc106f80 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86d9ffb700 1 -- 172.21.15.194:0/1839270407 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86d00042e0 con 0x7f86dc106f80 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.664+0000 7f86d9ffb700 1 -- 172.21.15.194:0/1839270407 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0011b10 con 0x7f86dc106f80 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 -- 172.21.15.194:0/1839270407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 msgr2=0x7f86dc109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 --2- 172.21.15.194:0/1839270407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc109370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86d00098a0 tx=0x7f86d0009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 -- 172.21.15.194:0/1839270407 shutdown_connections 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 --2- 172.21.15.194:0/1839270407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc109370 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 -- 172.21.15.194:0/1839270407 >> 172.21.15.194:0/1839270407 conn(0x7f86dc100930 msgr2=0x7f86dc102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:32.411 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 -- 172.21.15.194:0/1839270407 shutdown_connections 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.665+0000 7f86e15b8700 1 -- 172.21.15.194:0/1839270407 wait complete. 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86e15b8700 1 Processor -- start 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86e15b8700 1 -- start start 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86e15b8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86e15b8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc106cf0 con 0x7f86dc106f80 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86daffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86daffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:38382/0 (socket says 172.21.15.194:38382) 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86daffd700 1 -- 172.21.15.194:0/2471635714 learned_addr learned my addr 172.21.15.194:0/2471635714 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.666+0000 7f86daffd700 1 -- 172.21.15.194:0/2471635714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0009550 con 0x7f86dc106f80 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.667+0000 7f86daffd700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc1067b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f86d00117b0 tx=0x7f86d0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.667+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d00038b0 con 0x7f86dc106f80 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.667+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc104f30 con 0x7f86dc106f80 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.667+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86d0003a10 con 0x7f86dc106f80 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.667+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0024780 con 0x7f86dc106f80 2024-03-26T21:54:32.412 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.667+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc1053d0 con 0x7f86dc106f80 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.668+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f86d0024900 con 0x7f86dc106f80 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.668+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0623e0 con 0x7f86dc106f80 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.668+0000 7f86c3fff700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f86c4038720 0x7f86c403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.668+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f86d004d7e0 con 0x7f86dc106f80 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.668+0000 7f86da7fc700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f86c4038720 0x7f86c403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.668+0000 7f86da7fc700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f86c4038720 0x7f86c403abe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f86cc006fd0 tx=0x7f86cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.673+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f86d0024bb0 con 0x7f86dc106f80 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.898+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f86dc105a00 con 0x7f86c4038720 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.906+0000 7f86c3fff700 1 -- 172.21.15.194:0/2471635714 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f86dc105a00 con 0x7f86c4038720 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f86c4038720 msgr2=0x7f86c403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f86c4038720 0x7f86c403abe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f86cc006fd0 tx=0x7f86cc006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:32.413 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 msgr2=0x7f86dc1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc1067b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f86d00117b0 tx=0x7f86d0011890 comp rx=0 tx=0).stop 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 shutdown_connections 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f86c4038720 0x7f86c403abe0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 --2- 172.21.15.194:0/2471635714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86dc106f80 0x7f86dc1067b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.908+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 >> 172.21.15.194:0/2471635714 conn(0x7f86dc100930 msgr2=0x7f86dc1014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.909+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 shutdown_connections 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:31.909+0000 7f86e15b8700 1 -- 172.21.15.194:0/2471635714 wait complete. 2024-03-26T21:54:32.414 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying mgr service with default placement... 2024-03-26T21:54:32.519 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:32 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:32.520 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:32 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:32.520 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:32 smithi194 ceph-mon[29693]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:32.520 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:32 smithi194 ceph-mon[29693]: Saving service mon spec with placement count:5 2024-03-26T21:54:32.520 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:32 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.785+0000 7fa16544b700 1 Processor -- start 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.785+0000 7fa16544b700 1 -- start start 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa16544b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa160106960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa16544b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa160106f30 con 0x7fa160106540 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa15effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa160106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa15effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa160106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48886/0 (socket says 172.21.15.194:48886) 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa15effd700 1 -- 172.21.15.194:0/3507560949 learned_addr learned my addr 172.21.15.194:0/3507560949 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa15effd700 1 -- 172.21.15.194:0/3507560949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1601076a0 con 0x7fa160106540 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.786+0000 7fa15effd700 1 --2- 172.21.15.194:0/3507560949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa160106960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1500098a0 tx=0x7fa150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62110a3c912b97d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:33.986 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa15dffb700 1 -- 172.21.15.194:0/3507560949 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150004180 con 0x7fa160106540 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa15dffb700 1 -- 172.21.15.194:0/3507560949 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1500042e0 con 0x7fa160106540 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa15dffb700 1 -- 172.21.15.194:0/3507560949 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150011b10 con 0x7fa160106540 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa16544b700 1 -- 172.21.15.194:0/3507560949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 msgr2=0x7fa160106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa16544b700 1 --2- 172.21.15.194:0/3507560949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa160106960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1500098a0 tx=0x7fa150009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa16544b700 1 -- 172.21.15.194:0/3507560949 shutdown_connections 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.787+0000 7fa16544b700 1 --2- 172.21.15.194:0/3507560949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa160106960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.788+0000 7fa16544b700 1 -- 172.21.15.194:0/3507560949 >> 172.21.15.194:0/3507560949 conn(0x7fa160101b40 msgr2=0x7fa160103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.788+0000 7fa16544b700 1 -- 172.21.15.194:0/3507560949 shutdown_connections 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.788+0000 7fa16544b700 1 -- 172.21.15.194:0/3507560949 wait complete. 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.788+0000 7fa16544b700 1 Processor -- start 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.788+0000 7fa16544b700 1 -- start start 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa16544b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa16007bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa16544b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16007c040 con 0x7fa160106540 2024-03-26T21:54:33.987 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa15effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa16007bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa15effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa16007bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48892/0 (socket says 172.21.15.194:48892) 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa15effd700 1 -- 172.21.15.194:0/3551469109 learned_addr learned my addr 172.21.15.194:0/3551469109 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa15effd700 1 -- 172.21.15.194:0/3551469109 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150009550 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa15effd700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa16007bb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa1500117b0 tx=0x7fa150011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1500038b0 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa150003a10 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16007a280 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.789+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1500246e0 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.790+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16007a720 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.791+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fa15002d3f0 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.791+0000 7fa147fff700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa148038720 0x7fa14803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.791+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1600623e0 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.791+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa15004c840 con 0x7fa160106540 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.791+0000 7fa15e7fc700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa148038720 0x7fa14803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:33.988 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.791+0000 7fa15e7fc700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa148038720 0x7fa14803abe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa154006fd0 tx=0x7fa154006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:32.796+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa150010970 con 0x7fa160106540 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.014+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa16010a8a0 con 0x7fa148038720 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.018+0000 7fa147fff700 1 -- 172.21.15.194:0/3551469109 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa16010a8a0 con 0x7fa148038720 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa148038720 msgr2=0x7fa14803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa148038720 0x7fa14803abe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa154006fd0 tx=0x7fa154006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 msgr2=0x7fa16007bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa16007bb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa1500117b0 tx=0x7fa150011890 comp rx=0 tx=0).stop 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 shutdown_connections 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa148038720 0x7fa14803abe0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 --2- 172.21.15.194:0/3551469109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa160106540 0x7fa16007bb00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 >> 172.21.15.194:0/3551469109 conn(0x7fa160101b40 msgr2=0x7fa1601026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 shutdown_connections 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:33.020+0000 7fa16544b700 1 -- 172.21.15.194:0/3551469109 wait complete. 2024-03-26T21:54:33.989 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying crash service with default placement... 2024-03-26T21:54:34.107 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:34 smithi194 ceph-mon[29693]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:34.107 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:34 smithi194 ceph-mon[29693]: Saving service mgr spec with placement count:2 2024-03-26T21:54:34.107 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:34 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:35.243 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-26T21:54:35.243 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25b2f2700 1 Processor -- start 2024-03-26T21:54:35.243 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25b2f2700 1 -- start start 2024-03-26T21:54:35.243 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25b2f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2541057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:35.243 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25b2f2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff254105db0 con 0x7ff2541053c0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25908e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2541057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25908e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2541057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48908/0 (socket says 172.21.15.194:48908) 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.419+0000 7ff25908e700 1 -- 172.21.15.194:0/2901943652 learned_addr learned my addr 172.21.15.194:0/2901943652 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.420+0000 7ff25908e700 1 -- 172.21.15.194:0/2901943652 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2541065c0 con 0x7ff2541053c0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.420+0000 7ff25908e700 1 --2- 172.21.15.194:0/2901943652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2541057e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2500098a0 tx=0x7ff250009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1135fe8b9add4a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.420+0000 7ff24bfff700 1 -- 172.21.15.194:0/2901943652 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff250004180 con 0x7ff2541053c0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.420+0000 7ff24bfff700 1 -- 172.21.15.194:0/2901943652 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2500042e0 con 0x7ff2541053c0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.420+0000 7ff24bfff700 1 -- 172.21.15.194:0/2901943652 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff250011b10 con 0x7ff2541053c0 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 -- 172.21.15.194:0/2901943652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 msgr2=0x7ff2541057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 --2- 172.21.15.194:0/2901943652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2541057e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2500098a0 tx=0x7ff250009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:35.244 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 -- 172.21.15.194:0/2901943652 shutdown_connections 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 --2- 172.21.15.194:0/2901943652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2541057e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 -- 172.21.15.194:0/2901943652 >> 172.21.15.194:0/2901943652 conn(0x7ff2541009c0 msgr2=0x7ff254102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 -- 172.21.15.194:0/2901943652 shutdown_connections 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.421+0000 7ff25b2f2700 1 -- 172.21.15.194:0/2901943652 wait complete. 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25b2f2700 1 Processor -- start 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25b2f2700 1 -- start start 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25b2f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2540741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25b2f2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540746e0 con 0x7ff2541053c0 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25908e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2540741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25908e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2540741a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48918/0 (socket says 172.21.15.194:48918) 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25908e700 1 -- 172.21.15.194:0/4020205621 learned_addr learned my addr 172.21.15.194:0/4020205621 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25908e700 1 -- 172.21.15.194:0/4020205621 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250009550 con 0x7ff2541053c0 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff25908e700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2540741a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2500117b0 tx=0x7ff250011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2500038b0 con 0x7ff2541053c0 2024-03-26T21:54:35.245 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.422+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff250003a10 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.423+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2500246e0 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.423+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25419cf40 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.423+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25419d3e0 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.424+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7ff25002d3f0 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.424+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2541097f0 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.424+0000 7ff24a7fc700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff240038770 0x7ff24003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.424+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff25004c7f0 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.424+0000 7ff25888d700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff240038770 0x7ff24003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.424+0000 7ff25888d700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff240038770 0x7ff24003ac30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff244006fd0 tx=0x7ff244006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.429+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff250010c00 con 0x7ff2541053c0 2024-03-26T21:54:35.246 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.653+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7ff25419d6e0 con 0x7ff240038770 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.658+0000 7ff24a7fc700 1 -- 172.21.15.194:0/4020205621 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff25419d6e0 con 0x7ff240038770 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff240038770 msgr2=0x7ff24003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff240038770 0x7ff24003ac30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff244006fd0 tx=0x7ff244006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 msgr2=0x7ff2540741a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2540741a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2500117b0 tx=0x7ff250011890 comp rx=0 tx=0).stop 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 shutdown_connections 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7ff240038770 0x7ff24003ac30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 --2- 172.21.15.194:0/4020205621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2541053c0 0x7ff2540741a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 >> 172.21.15.194:0/4020205621 conn(0x7ff2541009c0 msgr2=0x7ff254101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 shutdown_connections 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:34.661+0000 7ff25b2f2700 1 -- 172.21.15.194:0/4020205621 wait complete. 2024-03-26T21:54:35.247 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying ceph-exporter service with default placement... 2024-03-26T21:54:35.361 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:35 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:35.361 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:35 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:36.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:36 smithi194 ceph-mon[29693]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:36.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:36 smithi194 ceph-mon[29693]: Saving service crash spec with placement * 2024-03-26T21:54:36.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:36 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:36.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:36 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:36.418 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f8a599700 1 Processor -- start 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f8a599700 1 -- start start 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f8a599700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f8a599700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f84106f00 con 0x7f3f84106510 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48920/0 (socket says 172.21.15.194:48920) 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.633+0000 7f3f83fff700 1 -- 172.21.15.194:0/3245376273 learned_addr learned my addr 172.21.15.194:0/3245376273 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.634+0000 7f3f83fff700 1 -- 172.21.15.194:0/3245376273 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f84107670 con 0x7f3f84106510 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.634+0000 7f3f83fff700 1 --2- 172.21.15.194:0/3245376273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84106930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f740098a0 tx=0x7f3f74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e99d0eb6e603dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.634+0000 7f3f82ffd700 1 -- 172.21.15.194:0/3245376273 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f74004180 con 0x7f3f84106510 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.634+0000 7f3f82ffd700 1 -- 172.21.15.194:0/3245376273 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f740042e0 con 0x7f3f84106510 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.634+0000 7f3f82ffd700 1 -- 172.21.15.194:0/3245376273 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f74011b10 con 0x7f3f84106510 2024-03-26T21:54:36.424 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 -- 172.21.15.194:0/3245376273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 msgr2=0x7f3f84106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 --2- 172.21.15.194:0/3245376273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84106930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f740098a0 tx=0x7f3f74009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 -- 172.21.15.194:0/3245376273 shutdown_connections 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 --2- 172.21.15.194:0/3245376273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84106930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 -- 172.21.15.194:0/3245376273 >> 172.21.15.194:0/3245376273 conn(0x7f3f84101b10 msgr2=0x7f3f84103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 -- 172.21.15.194:0/3245376273 shutdown_connections 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.635+0000 7f3f8a599700 1 -- 172.21.15.194:0/3245376273 wait complete. 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f8a599700 1 Processor -- start 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f8a599700 1 -- start start 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f8a599700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f8a599700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f84198550 con 0x7f3f84106510 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f83fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48924/0 (socket says 172.21.15.194:48924) 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f83fff700 1 -- 172.21.15.194:0/1821555529 learned_addr learned my addr 172.21.15.194:0/1821555529 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:36.425 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.636+0000 7f3f83fff700 1 -- 172.21.15.194:0/1821555529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f74009550 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.637+0000 7f3f83fff700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84198010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009fb0 tx=0x7f3f74011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.637+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f740038b0 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.637+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f74003a10 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.637+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f84198750 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.637+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f74019770 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.637+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f84198bf0 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.638+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f3f740198d0 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.638+0000 7f3f817fa700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3f6c03cb30 0x7f3f6c03eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.638+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3f7404d800 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.638+0000 7f3f837fe700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3f6c03cb30 0x7f3f6c03eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.638+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f70005320 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.638+0000 7f3f837fe700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3f6c03cb30 0x7f3f6c03eff0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006fd0 tx=0x7f3f78006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.643+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f74022d70 con 0x7f3f84106510 2024-03-26T21:54:36.426 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.863+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f70000bf0 con 0x7f3f6c03cb30 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.868+0000 7f3f817fa700 1 -- 172.21.15.194:0/1821555529 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f3f70000bf0 con 0x7f3f6c03cb30 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.870+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3f6c03cb30 msgr2=0x7f3f6c03eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.870+0000 7f3f8a599700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3f6c03cb30 0x7f3f6c03eff0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006fd0 tx=0x7f3f78006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.870+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 msgr2=0x7f3f84198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.870+0000 7f3f8a599700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84198010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009fb0 tx=0x7f3f74011890 comp rx=0 tx=0).stop 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.871+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 shutdown_connections 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.871+0000 7f3f8a599700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3f6c03cb30 0x7f3f6c03eff0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.871+0000 7f3f8a599700 1 --2- 172.21.15.194:0/1821555529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f84106510 0x7f3f84198010 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.871+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 >> 172.21.15.194:0/1821555529 conn(0x7f3f84101b10 msgr2=0x7f3f84102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.871+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 shutdown_connections 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:35.871+0000 7f3f8a599700 1 -- 172.21.15.194:0/1821555529 wait complete. 2024-03-26T21:54:36.427 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying prometheus service with default placement... 2024-03-26T21:54:37.332 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:37 smithi194 ceph-mon[29693]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:37.332 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:37 smithi194 ceph-mon[29693]: Saving service ceph-exporter spec with placement * 2024-03-26T21:54:37.332 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:37 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.806+0000 7f5978b58700 1 Processor -- start 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.806+0000 7f5978b58700 1 -- start start 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f5978b58700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f5974106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f5978b58700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974106f20 con 0x7f5974106530 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f5974106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f5974106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48928/0 (socket says 172.21.15.194:48928) 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597259c700 1 -- 172.21.15.194:0/1464662562 learned_addr learned my addr 172.21.15.194:0/1464662562 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597259c700 1 -- 172.21.15.194:0/1464662562 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5974107690 con 0x7f5974106530 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597259c700 1 --2- 172.21.15.194:0/1464662562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f5974106950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59640098a0 tx=0x7f5964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c618a7cbe27411 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:37.641 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597159a700 1 -- 172.21.15.194:0/1464662562 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5964004180 con 0x7f5974106530 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597159a700 1 -- 172.21.15.194:0/1464662562 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59640042e0 con 0x7f5974106530 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.807+0000 7f597159a700 1 -- 172.21.15.194:0/1464662562 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5964011b10 con 0x7f5974106530 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 -- 172.21.15.194:0/1464662562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 msgr2=0x7f5974106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 --2- 172.21.15.194:0/1464662562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f5974106950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59640098a0 tx=0x7f5964009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 -- 172.21.15.194:0/1464662562 shutdown_connections 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 --2- 172.21.15.194:0/1464662562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f5974106950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 -- 172.21.15.194:0/1464662562 >> 172.21.15.194:0/1464662562 conn(0x7f5974101b10 msgr2=0x7f5974103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 -- 172.21.15.194:0/1464662562 shutdown_connections 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.808+0000 7f5978b58700 1 -- 172.21.15.194:0/1464662562 wait complete. 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.809+0000 7f5978b58700 1 Processor -- start 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.809+0000 7f5978b58700 1 -- start start 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f5978b58700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f597419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f5978b58700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597419a760 con 0x7f5974106530 2024-03-26T21:54:37.642 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f597259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f597419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f597259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f597419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48930/0 (socket says 172.21.15.194:48930) 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f597259c700 1 -- 172.21.15.194:0/3817743983 learned_addr learned my addr 172.21.15.194:0/3817743983 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f597259c700 1 -- 172.21.15.194:0/3817743983 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964009550 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f597259c700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f597419a220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5964009fb0 tx=0x7f5964011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59640038b0 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5964003a10 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597419a960 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.810+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5964019770 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.811+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597419ae00 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.811+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f59640198d0 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.811+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597410a8c0 con 0x7f5974106530 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.812+0000 7f59637fe700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f595c0386e0 0x7f595c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:37.643 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.812+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f596404c920 con 0x7f5974106530 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.812+0000 7f5971d9b700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f595c0386e0 0x7f595c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.812+0000 7f5971d9b700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f595c0386e0 0x7f595c03aba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5968006fd0 tx=0x7f5968006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:36.817+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5964022b40 con 0x7f5974106530 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.025+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f597419d6f0 con 0x7f595c0386e0 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.029+0000 7f59637fe700 1 -- 172.21.15.194:0/3817743983 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f597419d6f0 con 0x7f595c0386e0 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.031+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f595c0386e0 msgr2=0x7f595c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.031+0000 7f5978b58700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f595c0386e0 0x7f595c03aba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5968006fd0 tx=0x7f5968006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.031+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 msgr2=0x7f597419a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.031+0000 7f5978b58700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f597419a220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5964009fb0 tx=0x7f5964011890 comp rx=0 tx=0).stop 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.032+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 shutdown_connections 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.032+0000 7f5978b58700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f595c0386e0 0x7f595c03aba0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.032+0000 7f5978b58700 1 --2- 172.21.15.194:0/3817743983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5974106530 0x7f597419a220 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.032+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 >> 172.21.15.194:0/3817743983 conn(0x7f5974101b10 msgr2=0x7f59741026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:37.644 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.032+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 shutdown_connections 2024-03-26T21:54:37.645 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:37.032+0000 7f5978b58700 1 -- 172.21.15.194:0/3817743983 wait complete. 2024-03-26T21:54:37.645 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying grafana service with default placement... 2024-03-26T21:54:38.279 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:38 smithi194 ceph-mon[29693]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:38.279 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:38 smithi194 ceph-mon[29693]: Saving service prometheus spec with placement count:1 2024-03-26T21:54:39.460 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-26T21:54:39.460 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.065+0000 7f220c71b700 1 Processor -- start 2024-03-26T21:54:39.460 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.065+0000 7f220c71b700 1 -- start start 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220c71b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f22041051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220c71b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2204079e50 con 0x7f2204104da0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220a4b7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f22041051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220a4b7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f22041051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48936/0 (socket says 172.21.15.194:48936) 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220a4b7700 1 -- 172.21.15.194:0/1090543102 learned_addr learned my addr 172.21.15.194:0/1090543102 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220a4b7700 1 -- 172.21.15.194:0/1090543102 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204105700 con 0x7f2204104da0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.066+0000 7f220a4b7700 1 --2- 172.21.15.194:0/1090543102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f22041051c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4cd45132db8534 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f22094b5700 1 -- 172.21.15.194:0/1090543102 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21f4004180 con 0x7f2204104da0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f22094b5700 1 -- 172.21.15.194:0/1090543102 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21f40042e0 con 0x7f2204104da0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f22094b5700 1 -- 172.21.15.194:0/1090543102 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21f4011b10 con 0x7f2204104da0 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 -- 172.21.15.194:0/1090543102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 msgr2=0x7f22041051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 --2- 172.21.15.194:0/1090543102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f22041051c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 -- 172.21.15.194:0/1090543102 shutdown_connections 2024-03-26T21:54:39.461 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 --2- 172.21.15.194:0/1090543102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f22041051c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 -- 172.21.15.194:0/1090543102 >> 172.21.15.194:0/1090543102 conn(0x7f2204100a00 msgr2=0x7f2204102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 -- 172.21.15.194:0/1090543102 shutdown_connections 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.067+0000 7f220c71b700 1 -- 172.21.15.194:0/1090543102 wait complete. 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220c71b700 1 Processor -- start 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220c71b700 1 -- start start 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220c71b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f220419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220c71b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220419c980 con 0x7f2204104da0 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220a4b7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f220419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220a4b7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f220419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48940/0 (socket says 172.21.15.194:48940) 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220a4b7700 1 -- 172.21.15.194:0/951486311 learned_addr learned my addr 172.21.15.194:0/951486311 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.068+0000 7f220a4b7700 1 -- 172.21.15.194:0/951486311 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f4009550 con 0x7f2204104da0 2024-03-26T21:54:39.462 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.069+0000 7f220a4b7700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f220419c440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21f40117b0 tx=0x7f21f4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.069+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21f40038b0 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.069+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21f4003a10 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.069+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220419cb80 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.069+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21f4024780 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.069+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220419d020 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.070+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f21f40248e0 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.070+0000 7f21fb7fe700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f21f0038770 0x7f21f003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.070+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22041960c0 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.070+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f21f404c750 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.070+0000 7f2209cb6700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f21f0038770 0x7f21f003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.070+0000 7f2209cb6700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f21f0038770 0x7f21f003ac30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2200006fd0 tx=0x7f2200006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.075+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f21f4024b90 con 0x7f2204104da0 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.286+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f220402cf20 con 0x7f21f0038770 2024-03-26T21:54:39.463 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.292+0000 7f21fb7fe700 1 -- 172.21.15.194:0/951486311 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f220402cf20 con 0x7f21f0038770 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f21f0038770 msgr2=0x7f21f003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f21f0038770 0x7f21f003ac30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2200006fd0 tx=0x7f2200006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 msgr2=0x7f220419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f220419c440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21f40117b0 tx=0x7f21f4011890 comp rx=0 tx=0).stop 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 shutdown_connections 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f21f0038770 0x7f21f003ac30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 --2- 172.21.15.194:0/951486311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2204104da0 0x7f220419c440 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 >> 172.21.15.194:0/951486311 conn(0x7f2204100a00 msgr2=0x7f2204193200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 shutdown_connections 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:38.294+0000 7f220c71b700 1 -- 172.21.15.194:0/951486311 wait complete. 2024-03-26T21:54:39.464 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying node-exporter service with default placement... 2024-03-26T21:54:39.583 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:39 smithi194 ceph-mon[29693]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:39.583 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:39 smithi194 ceph-mon[29693]: Saving service grafana spec with placement count:1 2024-03-26T21:54:39.583 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:39 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:40.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c947e700 1 Processor -- start 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c947e700 1 -- start start 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c947e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c41057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c947e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4105db0 con 0x7fa0c41053c0 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c41057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c41057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48952/0 (socket says 172.21.15.194:48952) 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c2ffd700 1 -- 172.21.15.194:0/3916956857 learned_addr learned my addr 172.21.15.194:0/3916956857 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c2ffd700 1 -- 172.21.15.194:0/3916956857 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c41065c0 con 0x7fa0c41053c0 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.880+0000 7fa0c2ffd700 1 --2- 172.21.15.194:0/3916956857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c41057e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80098a0 tx=0x7fa0b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e852466dc1d62497 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c1ffb700 1 -- 172.21.15.194:0/3916956857 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0b8004180 con 0x7fa0c41053c0 2024-03-26T21:54:40.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c1ffb700 1 -- 172.21.15.194:0/3916956857 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0b80042e0 con 0x7fa0c41053c0 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c1ffb700 1 -- 172.21.15.194:0/3916956857 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0b8011b10 con 0x7fa0c41053c0 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 -- 172.21.15.194:0/3916956857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 msgr2=0x7fa0c41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 --2- 172.21.15.194:0/3916956857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c41057e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80098a0 tx=0x7fa0b8009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 -- 172.21.15.194:0/3916956857 shutdown_connections 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 --2- 172.21.15.194:0/3916956857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c41057e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 -- 172.21.15.194:0/3916956857 >> 172.21.15.194:0/3916956857 conn(0x7fa0c41009c0 msgr2=0x7fa0c4102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 -- 172.21.15.194:0/3916956857 shutdown_connections 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.881+0000 7fa0c947e700 1 -- 172.21.15.194:0/3916956857 wait complete. 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c947e700 1 Processor -- start 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c947e700 1 -- start start 2024-03-26T21:54:40.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c947e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c4074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c947e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4074680 con 0x7fa0c41053c0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c4074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c4074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48956/0 (socket says 172.21.15.194:48956) 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.882+0000 7fa0c2ffd700 1 -- 172.21.15.194:0/3224939386 learned_addr learned my addr 172.21.15.194:0/3224939386 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0c2ffd700 1 -- 172.21.15.194:0/3224939386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8009550 con 0x7fa0c41053c0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0c2ffd700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c4074140 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80117b0 tx=0x7fa0b8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0b80038b0 con 0x7fa0c41053c0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0b8003a10 con 0x7fa0c41053c0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c419cfa0 con 0x7fa0c41053c0 2024-03-26T21:54:40.677 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0b80246e0 con 0x7fa0c41053c0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.883+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c419d3e0 con 0x7fa0c41053c0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.884+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fa0b802d3f0 con 0x7fa0c41053c0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.884+0000 7fa0abfff700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa0ac038770 0x7fa0ac03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.884+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c41097f0 con 0x7fa0c41053c0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.884+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa0b804c820 con 0x7fa0c41053c0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.884+0000 7fa0c27fc700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa0ac038770 0x7fa0ac03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.884+0000 7fa0c27fc700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa0ac038770 0x7fa0ac03ac30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4006fd0 tx=0x7fa0b4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:39.890+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa0b8010970 con 0x7fa0c41053c0 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.093+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0c419d6e0 con 0x7fa0ac038770 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.098+0000 7fa0abfff700 1 -- 172.21.15.194:0/3224939386 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fa0c419d6e0 con 0x7fa0ac038770 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa0ac038770 msgr2=0x7fa0ac03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa0ac038770 0x7fa0ac03ac30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4006fd0 tx=0x7fa0b4006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 msgr2=0x7fa0c4074140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c4074140 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80117b0 tx=0x7fa0b8011890 comp rx=0 tx=0).stop 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 shutdown_connections 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7fa0ac038770 0x7fa0ac03ac30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 --2- 172.21.15.194:0/3224939386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa0c41053c0 0x7fa0c4074140 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 >> 172.21.15.194:0/3224939386 conn(0x7fa0c41009c0 msgr2=0x7fa0c4101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 shutdown_connections 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:40.100+0000 7fa0c947e700 1 -- 172.21.15.194:0/3224939386 wait complete. 2024-03-26T21:54:40.679 INFO:teuthology.orchestra.run.smithi194.stdout:Deploying alertmanager service with default placement... 2024-03-26T21:54:41.316 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:41 smithi194 ceph-mon[29693]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:54:41.316 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:41 smithi194 ceph-mon[29693]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:41.316 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:41 smithi194 ceph-mon[29693]: Saving service node-exporter spec with placement * 2024-03-26T21:54:41.316 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:41 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:41.906 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bfa72f700 1 Processor -- start 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bfa72f700 1 -- start start 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bfa72f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf41051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bfa72f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4074590 con 0x7f9bf4104d80 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bf3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf41051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bf3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf41051a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48972/0 (socket says 172.21.15.194:48972) 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bf3fff700 1 -- 172.21.15.194:0/3065951643 learned_addr learned my addr 172.21.15.194:0/3065951643 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bf3fff700 1 -- 172.21.15.194:0/3065951643 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf41056e0 con 0x7f9bf4104d80 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.084+0000 7f9bf3fff700 1 --2- 172.21.15.194:0/3065951643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf41051a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9be40098a0 tx=0x7f9be4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b05cf4b28eae31 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bf2ffd700 1 -- 172.21.15.194:0/3065951643 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9be4004180 con 0x7f9bf4104d80 2024-03-26T21:54:41.907 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bf2ffd700 1 -- 172.21.15.194:0/3065951643 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9be40042e0 con 0x7f9bf4104d80 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bf2ffd700 1 -- 172.21.15.194:0/3065951643 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9be4011b10 con 0x7f9bf4104d80 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 -- 172.21.15.194:0/3065951643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 msgr2=0x7f9bf41051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 --2- 172.21.15.194:0/3065951643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf41051a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9be40098a0 tx=0x7f9be4009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 -- 172.21.15.194:0/3065951643 shutdown_connections 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 --2- 172.21.15.194:0/3065951643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf41051a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 -- 172.21.15.194:0/3065951643 >> 172.21.15.194:0/3065951643 conn(0x7f9bf41009c0 msgr2=0x7f9bf4102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 -- 172.21.15.194:0/3065951643 shutdown_connections 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.085+0000 7f9bfa72f700 1 -- 172.21.15.194:0/3065951643 wait complete. 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bfa72f700 1 Processor -- start 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bfa72f700 1 -- start start 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bfa72f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bfa72f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf419c960 con 0x7f9bf4104d80 2024-03-26T21:54:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bf3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bf3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48986/0 (socket says 172.21.15.194:48986) 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bf3fff700 1 -- 172.21.15.194:0/1947041864 learned_addr learned my addr 172.21.15.194:0/1947041864 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.086+0000 7f9bf3fff700 1 -- 172.21.15.194:0/1947041864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009550 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.087+0000 7f9bf3fff700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf419c420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9be40117b0 tx=0x7f9be4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.087+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9be40038b0 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.087+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9be4003a10 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.087+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9be4024780 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.087+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf419cb60 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.087+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf419d000 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.088+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f9be40248e0 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.088+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf4196150 con 0x7f9bf4104d80 2024-03-26T21:54:41.909 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.088+0000 7f9bf17fa700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f9bdc038770 0x7f9bdc03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.088+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9be404c970 con 0x7f9bf4104d80 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.088+0000 7f9bf37fe700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f9bdc038770 0x7f9bdc03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.088+0000 7f9bf37fe700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f9bdc038770 0x7f9bdc03ac30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006fd0 tx=0x7f9be8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.093+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9be4024b90 con 0x7f9bf4104d80 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.320+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bf402cf20 con 0x7f9bdc038770 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.326+0000 7f9bf17fa700 1 -- 172.21.15.194:0/1947041864 <== mgr.14118 v2:172.21.15.194:6800/549918706 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f9bf402cf20 con 0x7f9bdc038770 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f9bdc038770 msgr2=0x7f9bdc03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f9bdc038770 0x7f9bdc03ac30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006fd0 tx=0x7f9be8006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 msgr2=0x7f9bf419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf419c420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9be40117b0 tx=0x7f9be4011890 comp rx=0 tx=0).stop 2024-03-26T21:54:41.910 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 shutdown_connections 2024-03-26T21:54:41.911 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f9bdc038770 0x7f9bdc03ac30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:41.911 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 --2- 172.21.15.194:0/1947041864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bf4104d80 0x7f9bf419c420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:41.911 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 >> 172.21.15.194:0/1947041864 conn(0x7f9bf41009c0 msgr2=0x7f9bf4193290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:41.911 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 shutdown_connections 2024-03-26T21:54:41.911 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:41.328+0000 7f9bfa72f700 1 -- 172.21.15.194:0/1947041864 wait complete. 2024-03-26T21:54:42.594 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:42 smithi194 ceph-mon[29693]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:42.594 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:42 smithi194 ceph-mon[29693]: Saving service alertmanager spec with placement count:1 2024-03-26T21:54:42.594 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:42 smithi194 ceph-mon[29693]: from='mgr.14118 172.21.15.194:0/1445245637' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:42.595 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:42 smithi194 ceph-mon[29693]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:54:43.153 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.397+0000 7f4cdb95e700 1 Processor -- start 2024-03-26T21:54:43.153 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.397+0000 7f4cdb95e700 1 -- start start 2024-03-26T21:54:43.153 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cdb95e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd4109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cdb95e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40745f0 con 0x7f4cd4106f30 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cd96fa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd4109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cd96fa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd4109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48996/0 (socket says 172.21.15.194:48996) 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cd96fa700 1 -- 172.21.15.194:0/123984353 learned_addr learned my addr 172.21.15.194:0/123984353 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cd96fa700 1 -- 172.21.15.194:0/123984353 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd4109860 con 0x7f4cd4106f30 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4cd96fa700 1 --2- 172.21.15.194:0/123984353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd4109320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009e80 tx=0x7f4cd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d90c6be418e658d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4ccbfff700 1 -- 172.21.15.194:0/123984353 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cd000e070 con 0x7f4cd4106f30 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.398+0000 7f4ccbfff700 1 -- 172.21.15.194:0/123984353 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cd000b8c0 con 0x7f4cd4106f30 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4ccbfff700 1 -- 172.21.15.194:0/123984353 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cd00044d0 con 0x7f4cd4106f30 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 -- 172.21.15.194:0/123984353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 msgr2=0x7f4cd4109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:43.154 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 --2- 172.21.15.194:0/123984353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd4109320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009e80 tx=0x7f4cd000b330 comp rx=0 tx=0).stop 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 -- 172.21.15.194:0/123984353 shutdown_connections 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 --2- 172.21.15.194:0/123984353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd4109320 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 -- 172.21.15.194:0/123984353 >> 172.21.15.194:0/123984353 conn(0x7f4cd41009c0 msgr2=0x7f4cd4102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 -- 172.21.15.194:0/123984353 shutdown_connections 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.399+0000 7f4cdb95e700 1 -- 172.21.15.194:0/123984353 wait complete. 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cdb95e700 1 Processor -- start 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cdb95e700 1 -- start start 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cdb95e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cdb95e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd419a710 con 0x7f4cd4106f30 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cd96fa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cd96fa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:49006/0 (socket says 172.21.15.194:49006) 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cd96fa700 1 -- 172.21.15.194:0/1938045632 learned_addr learned my addr 172.21.15.194:0/1938045632 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cd96fa700 1 -- 172.21.15.194:0/1938045632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd0009b30 con 0x7f4cd4106f30 2024-03-26T21:54:43.155 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cd96fa700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd419a1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00072a0 tx=0x7f4cd000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cd000e040 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cd0003dc0 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.400+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd419a910 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.401+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cd0012750 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.401+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd419ae10 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.401+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f4cd0019040 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.401+0000 7f4cca7fc700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f4cc0038720 0x7f4cc003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.402+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4cd004c120 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.402+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd410a650 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.402+0000 7f4cd8ef9700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f4cc0038720 0x7f4cc003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.402+0000 7f4cd8ef9700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f4cc0038720 0x7f4cc003abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4006fd0 tx=0x7f4cc4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.406+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4cd002c940 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.598+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4cd410a870 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.602+0000 7f4cca7fc700 1 -- 172.21.15.194:0/1938045632 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4cd001f020 con 0x7f4cd4106f30 2024-03-26T21:54:43.156 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f4cc0038720 msgr2=0x7f4cc003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f4cc0038720 0x7f4cc003abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4006fd0 tx=0x7f4cc4006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 msgr2=0x7f4cd419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd419a1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00072a0 tx=0x7f4cd000be00 comp rx=0 tx=0).stop 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 shutdown_connections 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f4cc0038720 0x7f4cc003abe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 --2- 172.21.15.194:0/1938045632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cd4106f30 0x7f4cd419a1d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 >> 172.21.15.194:0/1938045632 conn(0x7f4cd41009c0 msgr2=0x7f4cd41014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 shutdown_connections 2024-03-26T21:54:43.157 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:42.605+0000 7f4cdb95e700 1 -- 172.21.15.194:0/1938045632 wait complete. 2024-03-26T21:54:43.757 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:43 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1938045632' entity='client.admin' 2024-03-26T21:54:44.344 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14abd7d700 1 Processor -- start 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14abd7d700 1 -- start start 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14abd7d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14abd7d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a407a820 con 0x7f14a407bde0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14a9b19700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14a9b19700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53830/0 (socket says 172.21.15.194:53830) 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.562+0000 7f14a9b19700 1 -- 172.21.15.194:0/117104297 learned_addr learned my addr 172.21.15.194:0/117104297 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.563+0000 7f14a9b19700 1 -- 172.21.15.194:0/117104297 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a407a960 con 0x7f14a407bde0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.563+0000 7f14a9b19700 1 --2- 172.21.15.194:0/117104297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a407a2e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f149c009e80 tx=0x7f149c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd2eacfb9231dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.563+0000 7f14a8b17700 1 -- 172.21.15.194:0/117104297 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f149c00e070 con 0x7f14a407bde0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.563+0000 7f14a8b17700 1 -- 172.21.15.194:0/117104297 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f149c00b8c0 con 0x7f14a407bde0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.563+0000 7f14a8b17700 1 -- 172.21.15.194:0/117104297 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f149c0044d0 con 0x7f14a407bde0 2024-03-26T21:54:44.345 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 -- 172.21.15.194:0/117104297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 msgr2=0x7f14a407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 --2- 172.21.15.194:0/117104297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a407a2e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f149c009e80 tx=0x7f149c00b330 comp rx=0 tx=0).stop 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 -- 172.21.15.194:0/117104297 shutdown_connections 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 --2- 172.21.15.194:0/117104297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a407a2e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 -- 172.21.15.194:0/117104297 >> 172.21.15.194:0/117104297 conn(0x7f14a4101b10 msgr2=0x7f14a4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 -- 172.21.15.194:0/117104297 shutdown_connections 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 -- 172.21.15.194:0/117104297 wait complete. 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 Processor -- start 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.564+0000 7f14abd7d700 1 -- start start 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14abd7d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14abd7d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a419c8e0 con 0x7f14a407bde0 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14a9b19700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14a9b19700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a419c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53838/0 (socket says 172.21.15.194:53838) 2024-03-26T21:54:44.346 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14a9b19700 1 -- 172.21.15.194:0/3192520816 learned_addr learned my addr 172.21.15.194:0/3192520816 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14a9b19700 1 -- 172.21.15.194:0/3192520816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c009b30 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14a9b19700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a419c3a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f149c0072a0 tx=0x7f149c00bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f149c00e040 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f149c003dc0 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a419cae0 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f149c012620 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.565+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a419cfe0 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.566+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f149c019040 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.566+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a410c850 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.566+0000 7f149affd700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f14900386c0 0x7f149003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.566+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f149c04c450 con 0x7f14a407bde0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.566+0000 7f14a9318700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f14900386c0 0x7f149003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.567+0000 7f14a9318700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f14900386c0 0x7f149003ab80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1494006fd0 tx=0x7f1494006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:44.347 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.571+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f149c02cb00 con 0x7f14a407bde0 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.762+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f14a410ca90 con 0x7f14a407bde0 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.768+0000 7f149affd700 1 -- 172.21.15.194:0/3192520816 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f149c01f070 con 0x7f14a407bde0 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f14900386c0 msgr2=0x7f149003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f14900386c0 0x7f149003ab80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1494006fd0 tx=0x7f1494006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 msgr2=0x7f14a419c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a419c3a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f149c0072a0 tx=0x7f149c00bf70 comp rx=0 tx=0).stop 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 shutdown_connections 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f14900386c0 0x7f149003ab80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 --2- 172.21.15.194:0/3192520816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14a407bde0 0x7f14a419c3a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 >> 172.21.15.194:0/3192520816 conn(0x7f14a4101b10 msgr2=0x7f14a41025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 shutdown_connections 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:43.770+0000 7f14abd7d700 1 -- 172.21.15.194:0/3192520816 wait complete. 2024-03-26T21:54:44.348 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling the dashboard module... 2024-03-26T21:54:44.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:44 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/3192520816' entity='client.admin' 2024-03-26T21:54:44.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:44 smithi194 ceph-mon[29693]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:54:46.083 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:45 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/4261762703' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T21:54:46.383 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e6efe700 1 Processor -- start 2024-03-26T21:54:46.383 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e6efe700 1 -- start start 2024-03-26T21:54:46.383 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e6efe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e0106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:46.383 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e6efe700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e0106f40 con 0x7f19e0106550 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e4c9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e0106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e4c9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e0106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53842/0 (socket says 172.21.15.194:53842) 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e4c9a700 1 -- 172.21.15.194:0/1030070363 learned_addr learned my addr 172.21.15.194:0/1030070363 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.788+0000 7f19e4c9a700 1 -- 172.21.15.194:0/1030070363 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e01076b0 con 0x7f19e0106550 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e4c9a700 1 --2- 172.21.15.194:0/1030070363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e0106970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19d40098a0 tx=0x7f19d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce84833781901c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19df7fe700 1 -- 172.21.15.194:0/1030070363 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d4004180 con 0x7f19e0106550 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19df7fe700 1 -- 172.21.15.194:0/1030070363 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19d40042e0 con 0x7f19e0106550 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19df7fe700 1 -- 172.21.15.194:0/1030070363 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d4011b10 con 0x7f19e0106550 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 -- 172.21.15.194:0/1030070363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 msgr2=0x7f19e0106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 --2- 172.21.15.194:0/1030070363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e0106970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19d40098a0 tx=0x7f19d4009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 -- 172.21.15.194:0/1030070363 shutdown_connections 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 --2- 172.21.15.194:0/1030070363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e0106970 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 -- 172.21.15.194:0/1030070363 >> 172.21.15.194:0/1030070363 conn(0x7f19e0101b10 msgr2=0x7f19e0103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:46.384 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 -- 172.21.15.194:0/1030070363 shutdown_connections 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.789+0000 7f19e6efe700 1 -- 172.21.15.194:0/1030070363 wait complete. 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e6efe700 1 Processor -- start 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e6efe700 1 -- start start 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e6efe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e007bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e6efe700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e007c040 con 0x7f19e0106550 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e4c9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e007bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e4c9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e007bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53858/0 (socket says 172.21.15.194:53858) 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e4c9a700 1 -- 172.21.15.194:0/4261762703 learned_addr learned my addr 172.21.15.194:0/4261762703 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.790+0000 7f19e4c9a700 1 -- 172.21.15.194:0/4261762703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4009550 con 0x7f19e0106550 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.791+0000 7f19e4c9a700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e007bb00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19d40117b0 tx=0x7f19d4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.791+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d40038b0 con 0x7f19e0106550 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.791+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19d4003a10 con 0x7f19e0106550 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.791+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e007a280 con 0x7f19e0106550 2024-03-26T21:54:46.385 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.791+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d40246e0 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.791+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e007a720 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.792+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f19d401a430 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.792+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e00623e0 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.792+0000 7f19ddffb700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f19d0038770 0x7f19d003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.792+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f19d404c7e0 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.792+0000 7f19dffff700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f19d0038770 0x7f19d003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.793+0000 7f19dffff700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f19d0038770 0x7f19d003ac30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f19cc006fd0 tx=0x7f19cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:44.797+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f19d4018b40 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.043+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f19e010a8b0 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.777+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f19d402d910 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.779+0000 7f19ddffb700 1 -- 172.21.15.194:0/4261762703 <== mon.0 v2:172.21.15.194:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f19d401ac30 con 0x7f19e0106550 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f19d0038770 msgr2=0x7f19d003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:46.386 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f19d0038770 0x7f19d003ac30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f19cc006fd0 tx=0x7f19cc006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 msgr2=0x7f19e007bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e007bb00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19d40117b0 tx=0x7f19d4011890 comp rx=0 tx=0).stop 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 shutdown_connections 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f19d0038770 0x7f19d003ac30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 --2- 172.21.15.194:0/4261762703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19e0106550 0x7f19e007bb00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 >> 172.21.15.194:0/4261762703 conn(0x7f19e0101b10 msgr2=0x7f19e0102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 shutdown_connections 2024-03-26T21:54:46.387 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:45.782+0000 7f19e6efe700 1 -- 172.21.15.194:0/4261762703 wait complete. 2024-03-26T21:54:47.067 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:46 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/4261762703' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T21:54:47.067 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:46 smithi194 ceph-mon[29693]: mgrmap e9: smithi194.wndbij(active, since 25s) 2024-03-26T21:54:47.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:47.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T21:54:47.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:54:47.757 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "active_name": "smithi194.wndbij", 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.866+0000 7f4949282700 1 Processor -- start 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.866+0000 7f4949282700 1 -- start start 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4949282700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f4944106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4949282700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4944106f00 con 0x7f4944106510 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4942ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f4944106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4942ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f4944106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53882/0 (socket says 172.21.15.194:53882) 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4942ffd700 1 -- 172.21.15.194:0/287745033 learned_addr learned my addr 172.21.15.194:0/287745033 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4942ffd700 1 -- 172.21.15.194:0/287745033 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49441076c0 con 0x7f4944106510 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.867+0000 7f4942ffd700 1 --2- 172.21.15.194:0/287745033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f4944106930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f49340098a0 tx=0x7f4934009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358876ef4e4fa9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:47.758 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4941ffb700 1 -- 172.21.15.194:0/287745033 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4934004180 con 0x7f4944106510 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4941ffb700 1 -- 172.21.15.194:0/287745033 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49340042e0 con 0x7f4944106510 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4941ffb700 1 -- 172.21.15.194:0/287745033 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4934011b10 con 0x7f4944106510 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 -- 172.21.15.194:0/287745033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 msgr2=0x7f4944106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 --2- 172.21.15.194:0/287745033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f4944106930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f49340098a0 tx=0x7f4934009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 -- 172.21.15.194:0/287745033 shutdown_connections 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 --2- 172.21.15.194:0/287745033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f4944106930 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 -- 172.21.15.194:0/287745033 >> 172.21.15.194:0/287745033 conn(0x7f4944101b10 msgr2=0x7f4944103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 -- 172.21.15.194:0/287745033 shutdown_connections 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.868+0000 7f4949282700 1 -- 172.21.15.194:0/287745033 wait complete. 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4949282700 1 Processor -- start 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4949282700 1 -- start start 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4949282700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f494419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:47.759 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4949282700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494419c970 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4942ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f494419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4942ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f494419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53888/0 (socket says 172.21.15.194:53888) 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.869+0000 7f4942ffd700 1 -- 172.21.15.194:0/1174923274 learned_addr learned my addr 172.21.15.194:0/1174923274 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f4942ffd700 1 -- 172.21.15.194:0/1174923274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934009550 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f4942ffd700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f494419c430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f49340117b0 tx=0x7f4934011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49340038b0 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4934003a10 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49340246e0 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494419cb70 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.870+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494419d010 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.871+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f493402d3f0 con 0x7f4944106510 2024-03-26T21:54:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.871+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494410a8f0 con 0x7f4944106510 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.871+0000 7f492bfff700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 0x7f492c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.871+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f493404c970 con 0x7f4944106510 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.871+0000 7f49427fc700 1 -- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 msgr2=0x7f492c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.871+0000 7f49427fc700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 0x7f492c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:46.876+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4934010de0 con 0x7f4944106510 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.072+0000 7f49427fc700 1 -- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 msgr2=0x7f492c03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.072+0000 7f49427fc700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 0x7f492c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.134+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f494419f8a0 con 0x7f4944106510 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.135+0000 7f492bfff700 1 -- 172.21.15.194:0/1174923274 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f4934010380 con 0x7f4944106510 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 msgr2=0x7f492c03ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 0x7f492c03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 msgr2=0x7f494419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:47.761 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f494419c430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f49340117b0 tx=0x7f4934011890 comp rx=0 tx=0).stop 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 shutdown_connections 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f492c038770 0x7f492c03ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 --2- 172.21.15.194:0/1174923274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4944106510 0x7f494419c430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 >> 172.21.15.194:0/1174923274 conn(0x7f4944101b10 msgr2=0x7f4944102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 shutdown_connections 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:47.137+0000 7f4949282700 1 -- 172.21.15.194:0/1174923274 wait complete. 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for the mgr to restart... 2024-03-26T21:54:47.762 INFO:teuthology.orchestra.run.smithi194.stdout:Waiting for mgr epoch 9... 2024-03-26T21:54:47.999 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:47 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1174923274' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T21:54:52.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: Active manager daemon smithi194.wndbij restarted 2024-03-26T21:54:52.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: Activating manager daemon smithi194.wndbij 2024-03-26T21:54:52.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T21:54:52.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: mgrmap e10: smithi194.wndbij(active, starting, since 0.00378787s) 2024-03-26T21:54:52.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mon metadata", "id": "smithi194"}]: dispatch 2024-03-26T21:54:52.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mgr metadata", "who": "smithi194.wndbij", "id": "smithi194.wndbij"}]: dispatch 2024-03-26T21:54:52.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:54:52.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:54:52.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:54:52.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: Manager daemon smithi194.wndbij is now available 2024-03-26T21:54:52.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:52 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:53.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:53 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi194.wndbij/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:54:53.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:53 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi194.wndbij/trash_purge_schedule"}]: dispatch 2024-03-26T21:54:53.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:53 smithi194 ceph-mon[29693]: mgrmap e11: smithi194.wndbij(active, since 1.00547s) 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.208+0000 7f3344f7b700 1 Processor -- start 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.208+0000 7f3344f7b700 1 -- start start 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.208+0000 7f3344f7b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f3340104be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f3344f7b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33401051b0 con 0x7f33401047c0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f3340104be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f3340104be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53896/0 (socket says 172.21.15.194:53896) 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333e59c700 1 -- 172.21.15.194:0/4178446201 learned_addr learned my addr 172.21.15.194:0/4178446201 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333e59c700 1 -- 172.21.15.194:0/4178446201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340107750 con 0x7f33401047c0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333e59c700 1 --2- 172.21.15.194:0/4178446201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f3340104be0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f33300098a0 tx=0x7f3330009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929238efb169659e server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333d59a700 1 -- 172.21.15.194:0/4178446201 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3330004180 con 0x7f33401047c0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333d59a700 1 -- 172.21.15.194:0/4178446201 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33300042e0 con 0x7f33401047c0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.209+0000 7f333d59a700 1 -- 172.21.15.194:0/4178446201 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3330011b10 con 0x7f33401047c0 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 -- 172.21.15.194:0/4178446201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 msgr2=0x7f3340104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 --2- 172.21.15.194:0/4178446201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f3340104be0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f33300098a0 tx=0x7f3330009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 -- 172.21.15.194:0/4178446201 shutdown_connections 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 --2- 172.21.15.194:0/4178446201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f3340104be0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 -- 172.21.15.194:0/4178446201 >> 172.21.15.194:0/4178446201 conn(0x7f3340100110 msgr2=0x7f3340102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 -- 172.21.15.194:0/4178446201 shutdown_connections 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.210+0000 7f3344f7b700 1 -- 172.21.15.194:0/4178446201 wait complete. 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f3344f7b700 1 Processor -- start 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f3344f7b700 1 -- start start 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f3344f7b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f334019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f3344f7b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334019a7b0 con 0x7f33401047c0 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f333e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f334019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f333e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f334019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:53912/0 (socket says 172.21.15.194:53912) 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.211+0000 7f333e59c700 1 -- 172.21.15.194:0/3537201356 learned_addr learned my addr 172.21.15.194:0/3537201356 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f333e59c700 1 -- 172.21.15.194:0/3537201356 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330009550 con 0x7f33401047c0 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f333e59c700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f334019a270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3330009fb0 tx=0x7f3330011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33300038b0 con 0x7f33401047c0 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334019a9b0 con 0x7f33401047c0 2024-03-26T21:54:54.218 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3330003a10 con 0x7f33401047c0 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3330019770 con 0x7f33401047c0 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.212+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334019ae50 con 0x7f33401047c0 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.213+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f33300198d0 con 0x7f33401047c0 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.213+0000 7f332f7fe700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.213+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 --> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332803b350 con 0x7f3328038780 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.213+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f333004c8d0 con 0x7f33401047c0 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.213+0000 7f333dd9b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 msgr2=0x7f332803ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.213+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.414+0000 7f333dd9b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 msgr2=0x7f332803ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.414+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.815+0000 7f333dd9b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 msgr2=0x7f332803ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:48.815+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:49.616+0000 7f333dd9b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 msgr2=0x7f332803ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:54.219 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:49.616+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:51.217+0000 7f333dd9b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 msgr2=0x7f332803ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.194:6800/549918706 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:51.217+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:52.603+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mgrmap(e 10) v1 ==== 45328+0+0 (secure 0 0 0) 0x7f333001ac10 con 0x7f33401047c0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:52.603+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 msgr2=0x7f332803ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:52.603+0000 7f332f7fe700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.604+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 11) v1 ==== 45455+0+0 (secure 0 0 0) 0x7f333002c3f0 con 0x7f33401047c0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.604+0000 7f332f7fe700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f332803bfd0 0x7f332803e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.605+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332803b350 con 0x7f332803bfd0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.605+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f332803bfd0 0x7f332803e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.605+0000 7f333dd9b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f332803bfd0 0x7f332803e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3334003a10 tx=0x7f3334009270 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.606+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f332803b350 con 0x7f332803bfd0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f33400623e0 con 0x7f332803bfd0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f332f7fe700 1 -- 172.21.15.194:0/3537201356 <== mgr.14162 v2:172.21.15.194:6800/3128608440 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f33400623e0 con 0x7f332803bfd0 2024-03-26T21:54:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f332803bfd0 msgr2=0x7f332803e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f332803bfd0 0x7f332803e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3334003a10 tx=0x7f3334009270 comp rx=0 tx=0).stop 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 msgr2=0x7f334019a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f334019a270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3330009fb0 tx=0x7f3330011890 comp rx=0 tx=0).stop 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 shutdown_connections 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f332803bfd0 0x7f332803e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33401047c0 0x7f334019a270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 --2- 172.21.15.194:0/3537201356 >> [v2:172.21.15.194:6800/549918706,v1:172.21.15.194:6801/549918706] conn(0x7f3328038780 0x7f332803ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 >> 172.21.15.194:0/3537201356 conn(0x7f3340100110 msgr2=0x7f3340100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.612+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 shutdown_connections 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:53.613+0000 7f3344f7b700 1 -- 172.21.15.194:0/3537201356 wait complete. 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:mgr epoch 9 is available 2024-03-26T21:54:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Generating a dashboard self-signed certificate... 2024-03-26T21:54:54.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:54 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:52] ENGINE Bus STARTING 2024-03-26T21:54:54.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:54 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:53] ENGINE Serving on https://172.21.15.194:7150 2024-03-26T21:54:54.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:54 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:53] ENGINE Serving on http://172.21.15.194:8765 2024-03-26T21:54:54.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:54 smithi194 ceph-mon[29693]: [26/Mar/2024:21:54:53] ENGINE Bus STARTED 2024-03-26T21:54:54.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:54 smithi194 ceph-mon[29693]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T21:54:54.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:54 smithi194 ceph-mon[29693]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.733+0000 7fd586a41700 1 Processor -- start 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.733+0000 7fd586a41700 1 -- start start 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd586a41700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd5801068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd586a41700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd580106ec0 con 0x7fd5801064d0 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd5801068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd5801068f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42168/0 (socket says 172.21.15.194:42168) 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57ffff700 1 -- 172.21.15.194:0/4064513246 learned_addr learned my addr 172.21.15.194:0/4064513246 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57ffff700 1 -- 172.21.15.194:0/4064513246 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5801076d0 con 0x7fd5801064d0 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57ffff700 1 --2- 172.21.15.194:0/4064513246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd5801068f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5740098a0 tx=0x7fd574009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7031ef70fbf7c4fe server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57effd700 1 -- 172.21.15.194:0/4064513246 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd574004180 con 0x7fd5801064d0 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57effd700 1 -- 172.21.15.194:0/4064513246 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5740042e0 con 0x7fd5801064d0 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.734+0000 7fd57effd700 1 -- 172.21.15.194:0/4064513246 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd574011b10 con 0x7fd5801064d0 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 -- 172.21.15.194:0/4064513246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 msgr2=0x7fd5801068f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 --2- 172.21.15.194:0/4064513246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd5801068f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5740098a0 tx=0x7fd574009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 -- 172.21.15.194:0/4064513246 shutdown_connections 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 --2- 172.21.15.194:0/4064513246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd5801068f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 -- 172.21.15.194:0/4064513246 >> 172.21.15.194:0/4064513246 conn(0x7fd580101a50 msgr2=0x7fd580103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 -- 172.21.15.194:0/4064513246 shutdown_connections 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.735+0000 7fd586a41700 1 -- 172.21.15.194:0/4064513246 wait complete. 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd586a41700 1 Processor -- start 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd586a41700 1 -- start start 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd586a41700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd58019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:55.673 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd586a41700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58019a7b0 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd57ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd58019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd57ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd58019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42174/0 (socket says 172.21.15.194:42174) 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd57ffff700 1 -- 172.21.15.194:0/3742090076 learned_addr learned my addr 172.21.15.194:0/3742090076 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd57ffff700 1 -- 172.21.15.194:0/3742090076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd574009550 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.736+0000 7fd57ffff700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd58019a270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5740117b0 tx=0x7fd574011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.737+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5740038b0 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.737+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd574003a10 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.737+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58019a9b0 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.737+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5740246e0 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.737+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58019ae50 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.738+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 11) v1 ==== 45455+0+0 (secure 0 0 0) 0x7fd57401ab80 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.738+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58010a900 con 0x7fd5801064d0 2024-03-26T21:54:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.738+0000 7fd57d7fa700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd5680386a0 0x7fd56803ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.738+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd57404c850 con 0x7fd5801064d0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.738+0000 7fd57f7fe700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd5680386a0 0x7fd56803ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.738+0000 7fd57f7fe700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd5680386a0 0x7fd56803ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd570006fd0 tx=0x7fd570006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.743+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd574010380 con 0x7fd5801064d0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.960+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd58019d740 con 0x7fd5680386a0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:54.988+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7fd57402d770 con 0x7fd5801064d0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.052+0000 7fd57d7fa700 1 -- 172.21.15.194:0/3742090076 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd58019d740 con 0x7fd5680386a0 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd5680386a0 msgr2=0x7fd56803ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd5680386a0 0x7fd56803ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd570006fd0 tx=0x7fd570006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 msgr2=0x7fd58019a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd58019a270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5740117b0 tx=0x7fd574011890 comp rx=0 tx=0).stop 2024-03-26T21:54:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 shutdown_connections 2024-03-26T21:54:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd5680386a0 0x7fd56803ab60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 --2- 172.21.15.194:0/3742090076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5801064d0 0x7fd58019a270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 >> 172.21.15.194:0/3742090076 conn(0x7fd580101a50 msgr2=0x7fd580102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 shutdown_connections 2024-03-26T21:54:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:55.055+0000 7fd586a41700 1 -- 172.21.15.194:0/3742090076 wait complete. 2024-03-26T21:54:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:Creating initial admin user... 2024-03-26T21:54:55.975 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:55 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:55.976 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:55 smithi194 ceph-mon[29693]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:55.976 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:55 smithi194 ceph-mon[29693]: mgrmap e12: smithi194.wndbij(active, since 2s) 2024-03-26T21:54:55.976 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:55 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:55.976 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:55 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:57.128 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$AK5x41hyJqhIKrY.9HZerOeiWEv5faHsL/jVLI0U4JEOMGm4uHfoe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711490096, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T21:54:57.128 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.082+0000 7f1db93af700 1 Processor -- start 2024-03-26T21:54:57.128 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.082+0000 7f1db93af700 1 -- start start 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.082+0000 7f1db93af700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4104c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.082+0000 7f1db93af700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41051e0 con 0x7f1db41047f0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4104c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4104c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42186/0 (socket says 172.21.15.194:42186) 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db2ffd700 1 -- 172.21.15.194:0/3642754625 learned_addr learned my addr 172.21.15.194:0/3642754625 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db2ffd700 1 -- 172.21.15.194:0/3642754625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db4105320 con 0x7f1db41047f0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db2ffd700 1 --2- 172.21.15.194:0/3642754625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4104c10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1da40098a0 tx=0x7f1da4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2406340d96a6c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db27fc700 1 -- 172.21.15.194:0/3642754625 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da4004180 con 0x7f1db41047f0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db27fc700 1 -- 172.21.15.194:0/3642754625 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1da40042e0 con 0x7f1db41047f0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.083+0000 7f1db27fc700 1 -- 172.21.15.194:0/3642754625 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da4011b10 con 0x7f1db41047f0 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 -- 172.21.15.194:0/3642754625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 msgr2=0x7f1db4104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 --2- 172.21.15.194:0/3642754625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4104c10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1da40098a0 tx=0x7f1da4009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 -- 172.21.15.194:0/3642754625 shutdown_connections 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 --2- 172.21.15.194:0/3642754625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4104c10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 -- 172.21.15.194:0/3642754625 >> 172.21.15.194:0/3642754625 conn(0x7f1db4100170 msgr2=0x7f1db4102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 -- 172.21.15.194:0/3642754625 shutdown_connections 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 -- 172.21.15.194:0/3642754625 wait complete. 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.084+0000 7f1db93af700 1 Processor -- start 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db93af700 1 -- start start 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db93af700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db93af700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db4107590 con 0x7f1db41047f0 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42192/0 (socket says 172.21.15.194:42192) 2024-03-26T21:54:57.130 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db2ffd700 1 -- 172.21.15.194:0/2315311978 learned_addr learned my addr 172.21.15.194:0/2315311978 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db2ffd700 1 -- 172.21.15.194:0/2315311978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009550 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db2ffd700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4107050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009fb0 tx=0x7f1da4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da40038b0 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1da4003a10 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db4105820 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da4019770 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.085+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db4105cc0 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.086+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f1da401a430 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.086+0000 7f1db0ff9700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1da0038780 0x7f1da003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.086+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1da404c960 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.086+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db40623e0 con 0x7f1db41047f0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.087+0000 7f1daa7fc700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1da0038780 0x7f1da003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:57.131 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.087+0000 7f1daa7fc700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1da0038780 0x7f1da003ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c006fd0 tx=0x7f1d9c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.091+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1da402c3a0 con 0x7f1db41047f0 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.304+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1db410aaa0 con 0x7f1da0038780 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.530+0000 7f1db0ff9700 1 -- 172.21.15.194:0/2315311978 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1d9c009e20 con 0x7f1da0038780 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1da0038780 msgr2=0x7f1da003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1da0038780 0x7f1da003ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c006fd0 tx=0x7f1d9c006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 msgr2=0x7f1db4107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4107050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009fb0 tx=0x7f1da4011890 comp rx=0 tx=0).stop 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 shutdown_connections 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1da0038780 0x7f1da003ac40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 --2- 172.21.15.194:0/2315311978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1db41047f0 0x7f1db4107050 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 >> 172.21.15.194:0/2315311978 conn(0x7f1db4100170 msgr2=0x7f1db4100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 shutdown_connections 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:56.532+0000 7f1db93af700 1 -- 172.21.15.194:0/2315311978 wait complete. 2024-03-26T21:54:57.132 INFO:teuthology.orchestra.run.smithi194.stdout:Fetching dashboard port number... 2024-03-26T21:54:57.456 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:57 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:57.456 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:57 smithi194 ceph-mon[29693]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:54:57.456 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:57 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:54:58.795 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:54:58 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1579312997' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T21:54:59.267 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T21:54:59.267 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.265+0000 7f2291ef2700 1 Processor -- start 2024-03-26T21:54:59.267 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.265+0000 7f2291ef2700 1 -- start start 2024-03-26T21:54:59.267 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.265+0000 7f2291ef2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:59.267 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.265+0000 7f2291ef2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c106ef0 con 0x7f228c106500 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42196/0 (socket says 172.21.15.194:42196) 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228b7fe700 1 -- 172.21.15.194:0/351963620 learned_addr learned my addr 172.21.15.194:0/351963620 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228b7fe700 1 -- 172.21.15.194:0/351963620 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c1076b0 con 0x7f228c106500 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228b7fe700 1 --2- 172.21.15.194:0/351963620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c106920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f227c0098a0 tx=0x7f227c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f67e311bd3c4842 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228a7fc700 1 -- 172.21.15.194:0/351963620 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227c004180 con 0x7f228c106500 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228a7fc700 1 -- 172.21.15.194:0/351963620 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f227c0042e0 con 0x7f228c106500 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.266+0000 7f228a7fc700 1 -- 172.21.15.194:0/351963620 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227c011b10 con 0x7f228c106500 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 -- 172.21.15.194:0/351963620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 msgr2=0x7f228c106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 --2- 172.21.15.194:0/351963620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c106920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f227c0098a0 tx=0x7f227c009bb0 comp rx=0 tx=0).stop 2024-03-26T21:54:59.268 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 -- 172.21.15.194:0/351963620 shutdown_connections 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 --2- 172.21.15.194:0/351963620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c106920 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 -- 172.21.15.194:0/351963620 >> 172.21.15.194:0/351963620 conn(0x7f228c101b40 msgr2=0x7f228c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 -- 172.21.15.194:0/351963620 shutdown_connections 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 -- 172.21.15.194:0/351963620 wait complete. 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 Processor -- start 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.267+0000 7f2291ef2700 1 -- start start 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2291ef2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2291ef2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c19a760 con 0x7f228c106500 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f228b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f228b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42202/0 (socket says 172.21.15.194:42202) 2024-03-26T21:54:59.269 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f228b7fe700 1 -- 172.21.15.194:0/1579312997 learned_addr learned my addr 172.21.15.194:0/1579312997 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f228b7fe700 1 -- 172.21.15.194:0/1579312997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c009550 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f228b7fe700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c19a220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f227c009fb0 tx=0x7f227c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227c0038b0 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f227c003a10 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c19a960 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227c019770 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.268+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c19ae00 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.269+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f227c0198d0 con 0x7f228c106500 2024-03-26T21:54:59.270 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.269+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c10a8e0 con 0x7f228c106500 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.269+0000 7f2288ff9700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2274038780 0x7f227403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.269+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f227c04ca50 con 0x7f228c106500 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.269+0000 7f228affd700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2274038780 0x7f227403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.270+0000 7f228affd700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2274038780 0x7f227403ac40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2280006fd0 tx=0x7f2280006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.274+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f227c0224f0 con 0x7f228c106500 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.475+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f228c07a0c0 con 0x7f228c106500 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.475+0000 7f2288ff9700 1 -- 172.21.15.194:0/1579312997 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f227c022310 con 0x7f228c106500 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2274038780 msgr2=0x7f227403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2274038780 0x7f227403ac40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2280006fd0 tx=0x7f2280006e40 comp rx=0 tx=0).stop 2024-03-26T21:54:59.271 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 msgr2=0x7f228c19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c19a220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f227c009fb0 tx=0x7f227c011890 comp rx=0 tx=0).stop 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 shutdown_connections 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2274038780 0x7f227403ac40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 --2- 172.21.15.194:0/1579312997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f228c106500 0x7f228c19a220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.477+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 >> 172.21.15.194:0/1579312997 conn(0x7f228c101b40 msgr2=0x7f228c1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.478+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 shutdown_connections 2024-03-26T21:54:59.272 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:58.478+0000 7f2291ef2700 1 -- 172.21.15.194:0/1579312997 wait complete. 2024-03-26T21:54:59.273 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:54:59.273 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout disabled 2024-03-26T21:54:59.279 INFO:teuthology.orchestra.run.smithi194.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:54:59.279 INFO:teuthology.orchestra.run.smithi194.stdout:systemctl: stdout inactive 2024-03-26T21:54:59.279 INFO:teuthology.orchestra.run.smithi194.stdout:firewalld.service is not enabled 2024-03-26T21:54:59.279 INFO:teuthology.orchestra.run.smithi194.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout:Ceph Dashboard is now available at: 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout: URL: https://smithi194.front.sepia.ceph.com:8443/ 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout: User: admin 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout: Password: oxm6bjinn0 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:54:59.281 INFO:teuthology.orchestra.run.smithi194.stdout:Saving cluster configuration to /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/config directory 2024-03-26T21:54:59.282 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling autotune for osd_memory_target 2024-03-26T21:55:00.541 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.754+0000 7f2169590700 1 Processor -- start 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.754+0000 7f2169590700 1 -- start start 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.754+0000 7f2169590700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f2164109400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.754+0000 7f2169590700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2164074510 con 0x7f2164107010 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f2164109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f2164109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42218/0 (socket says 172.21.15.194:42218) 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2162ffd700 1 -- 172.21.15.194:0/766356000 learned_addr learned my addr 172.21.15.194:0/766356000 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2162ffd700 1 -- 172.21.15.194:0/766356000 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164074650 con 0x7f2164107010 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2162ffd700 1 --2- 172.21.15.194:0/766356000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f2164109400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21540098a0 tx=0x7f2154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c8363dd5a37a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2161ffb700 1 -- 172.21.15.194:0/766356000 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2154004180 con 0x7f2164107010 2024-03-26T21:55:00.542 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2161ffb700 1 -- 172.21.15.194:0/766356000 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21540042e0 con 0x7f2164107010 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.755+0000 7f2161ffb700 1 -- 172.21.15.194:0/766356000 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2154011b10 con 0x7f2164107010 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 -- 172.21.15.194:0/766356000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 msgr2=0x7f2164109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 --2- 172.21.15.194:0/766356000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f2164109400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21540098a0 tx=0x7f2154009bb0 comp rx=0 tx=0).stop 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 -- 172.21.15.194:0/766356000 shutdown_connections 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 --2- 172.21.15.194:0/766356000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f2164109400 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 -- 172.21.15.194:0/766356000 >> 172.21.15.194:0/766356000 conn(0x7f21641009c0 msgr2=0x7f2164102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 -- 172.21.15.194:0/766356000 shutdown_connections 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 -- 172.21.15.194:0/766356000 wait complete. 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.756+0000 7f2169590700 1 Processor -- start 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2169590700 1 -- start start 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2169590700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f216419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2169590700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216419a7b0 con 0x7f2164107010 2024-03-26T21:55:00.543 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f216419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f216419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42222/0 (socket says 172.21.15.194:42222) 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2162ffd700 1 -- 172.21.15.194:0/1589231664 learned_addr learned my addr 172.21.15.194:0/1589231664 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2162ffd700 1 -- 172.21.15.194:0/1589231664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2154009550 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2162ffd700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f216419a270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2154009fb0 tx=0x7f2154011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21540038b0 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2154003a10 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216419a9b0 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2154019750 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.757+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216419ae50 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.758+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f215401ac10 con 0x7f2164107010 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.758+0000 7f214bfff700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f214c038820 0x7f214c03ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:00.544 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.758+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216410a660 con 0x7f2164107010 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.759+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f215404ca20 con 0x7f2164107010 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.759+0000 7f21627fc700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f214c038820 0x7f214c03ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.759+0000 7f21627fc700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f214c038820 0x7f214c03ace0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2158006fd0 tx=0x7f2158006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.763+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2154022700 con 0x7f2164107010 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.956+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f216410a8a0 con 0x7f2164107010 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.956+0000 7f214bfff700 1 -- 172.21.15.194:0/1589231664 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2154022310 con 0x7f2164107010 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f214c038820 msgr2=0x7f214c03ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f214c038820 0x7f214c03ace0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2158006fd0 tx=0x7f2158006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 msgr2=0x7f216419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f216419a270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2154009fb0 tx=0x7f2154011890 comp rx=0 tx=0).stop 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 shutdown_connections 2024-03-26T21:55:00.545 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f214c038820 0x7f214c03ace0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:00.546 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 --2- 172.21.15.194:0/1589231664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2164107010 0x7f216419a270 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:00.546 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 >> 172.21.15.194:0/1589231664 conn(0x7f21641009c0 msgr2=0x7f21641014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:00.546 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 shutdown_connections 2024-03-26T21:55:00.546 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:54:59.958+0000 7f2169590700 1 -- 172.21.15.194:0/1589231664 wait complete. 2024-03-26T21:55:00.799 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:00 smithi194 ceph-mon[29693]: mgrmap e13: smithi194.wndbij(active, since 6s) 2024-03-26T21:55:01.889 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56edd89700 1 Processor -- start 2024-03-26T21:55:01.889 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56edd89700 1 -- start start 2024-03-26T21:55:01.889 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56edd89700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e8106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:01.889 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56edd89700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e8106f40 con 0x7f56e8106550 2024-03-26T21:55:01.889 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e8106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e8106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42230/0 (socket says 172.21.15.194:42230) 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56e77fe700 1 -- 172.21.15.194:0/2715608347 learned_addr learned my addr 172.21.15.194:0/2715608347 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.996+0000 7f56e77fe700 1 -- 172.21.15.194:0/2715608347 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e81076b0 con 0x7f56e8106550 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56e77fe700 1 --2- 172.21.15.194:0/2715608347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e8106970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56d80098a0 tx=0x7f56d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7448b80f5336e281 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56e67fc700 1 -- 172.21.15.194:0/2715608347 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d8004180 con 0x7f56e8106550 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56e67fc700 1 -- 172.21.15.194:0/2715608347 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56d80042e0 con 0x7f56e8106550 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56e67fc700 1 -- 172.21.15.194:0/2715608347 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d8011b10 con 0x7f56e8106550 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 -- 172.21.15.194:0/2715608347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 msgr2=0x7f56e8106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 --2- 172.21.15.194:0/2715608347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e8106970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56d80098a0 tx=0x7f56d8009bb0 comp rx=0 tx=0).stop 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 -- 172.21.15.194:0/2715608347 shutdown_connections 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 --2- 172.21.15.194:0/2715608347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e8106970 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 -- 172.21.15.194:0/2715608347 >> 172.21.15.194:0/2715608347 conn(0x7f56e8101b10 msgr2=0x7f56e8103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 -- 172.21.15.194:0/2715608347 shutdown_connections 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.997+0000 7f56edd89700 1 -- 172.21.15.194:0/2715608347 wait complete. 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56edd89700 1 Processor -- start 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56edd89700 1 -- start start 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56edd89700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56edd89700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e819a770 con 0x7f56e8106550 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e819a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42232/0 (socket says 172.21.15.194:42232) 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.998+0000 7f56e77fe700 1 -- 172.21.15.194:0/981023498 learned_addr learned my addr 172.21.15.194:0/981023498 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56e77fe700 1 -- 172.21.15.194:0/981023498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d8009550 con 0x7f56e8106550 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56e77fe700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e819a230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f56d80117b0 tx=0x7f56d8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d80038b0 con 0x7f56e8106550 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56d8003a10 con 0x7f56e8106550 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d80246e0 con 0x7f56e8106550 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e819a970 con 0x7f56e8106550 2024-03-26T21:55:01.891 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:00.999+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e819ae10 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.000+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f56d802d3f0 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.000+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e810a8e0 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.000+0000 7f56e4ff9700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56d0038860 0x7f56d003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.000+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f56d804ca00 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.000+0000 7f56e6ffd700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56d0038860 0x7f56d003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.001+0000 7f56e6ffd700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56d0038860 0x7f56d003ad20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f56dc006fd0 tx=0x7f56dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.005+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f56d8010c00 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.307+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f56e819d700 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.309+0000 7f56e4ff9700 1 -- 172.21.15.194:0/981023498 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f56d80107f0 con 0x7f56e8106550 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56d0038860 msgr2=0x7f56d003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56d0038860 0x7f56d003ad20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f56dc006fd0 tx=0x7f56dc006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 msgr2=0x7f56e819a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e819a230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f56d80117b0 tx=0x7f56d8011890 comp rx=0 tx=0).stop 2024-03-26T21:55:01.892 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 shutdown_connections 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56d0038860 0x7f56d003ad20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 --2- 172.21.15.194:0/981023498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56e8106550 0x7f56e819a230 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 >> 172.21.15.194:0/981023498 conn(0x7f56e8101b10 msgr2=0x7f56e8102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 shutdown_connections 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr 2024-03-26T21:55:01.311+0000 7f56edd89700 1 -- 172.21.15.194:0/981023498 wait complete. 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T21:55:01.893 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout: ceph telemetry on 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout:For more information see: 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:01.894 INFO:teuthology.orchestra.run.smithi194.stdout:Bootstrap complete. 2024-03-26T21:55:01.927 INFO:tasks.cephadm:Fetching config... 2024-03-26T21:55:01.927 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:55:01.927 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T21:55:01.983 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T21:55:01.983 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:55:01.984 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T21:55:02.039 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T21:55:02.039 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:55:02.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/mon.smithi194/keyring of=/dev/stdout 2024-03-26T21:55:02.105 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T21:55:02.105 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-26T21:55:02.106 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T21:55:02.161 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T21:55:02.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ppzNzLTpeplBS4fNfSgIWijhpJVnSscHATdj7ONvLCoKNYlNoGzDnJRht9erX01bWiOH4Z/bWqFOS2RAq1d50EE1N2NYCF3NA4i9iWR5J6Wnj8tdiRecO9/j/am+MUl0URPMiKUjqlSk8nxgD3CmJkouv7+em8Mv9eVVZwGQqiYXoNFVfwjNIYD1IGBhW0zzCoM0eZ+tdnUy7wUD1nWOXtPtB63tr4Yek9d9poVlVGmidAs6Y8JYxW9Bj3kaJgmHPEJwpTUXTBi3e7cR224JC17YeXV20PNAQ2oBMSuZDxBkYEE0hVVeVHagiZvu6op24VxoZYt/qQi1CDCE8kZQf2edYhwANDmRUFFhTtaYojjEihgQj1yBaG+pNmK8vv+A+1hn+oHszbKhEBaonliYPctqt/YS12YJj7cXv6v2g/8OzO54xBt0+0cQouyAHfZbbOckOgU1yt5fNGFyvF+JvAAB01d3Hiwtwk8sMBrSJPATkOcRaWdHBzS6fXyOCmM= ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T21:55:02.242 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ppzNzLTpeplBS4fNfSgIWijhpJVnSscHATdj7ONvLCoKNYlNoGzDnJRht9erX01bWiOH4Z/bWqFOS2RAq1d50EE1N2NYCF3NA4i9iWR5J6Wnj8tdiRecO9/j/am+MUl0URPMiKUjqlSk8nxgD3CmJkouv7+em8Mv9eVVZwGQqiYXoNFVfwjNIYD1IGBhW0zzCoM0eZ+tdnUy7wUD1nWOXtPtB63tr4Yek9d9poVlVGmidAs6Y8JYxW9Bj3kaJgmHPEJwpTUXTBi3e7cR224JC17YeXV20PNAQ2oBMSuZDxBkYEE0hVVeVHagiZvu6op24VxoZYt/qQi1CDCE8kZQf2edYhwANDmRUFFhTtaYojjEihgQj1yBaG+pNmK8vv+A+1hn+oHszbKhEBaonliYPctqt/YS12YJj7cXv6v2g/8OzO54xBt0+0cQouyAHfZbbOckOgU1yt5fNGFyvF+JvAAB01d3Hiwtwk8sMBrSJPATkOcRaWdHBzS6fXyOCmM= ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:55:02.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ppzNzLTpeplBS4fNfSgIWijhpJVnSscHATdj7ONvLCoKNYlNoGzDnJRht9erX01bWiOH4Z/bWqFOS2RAq1d50EE1N2NYCF3NA4i9iWR5J6Wnj8tdiRecO9/j/am+MUl0URPMiKUjqlSk8nxgD3CmJkouv7+em8Mv9eVVZwGQqiYXoNFVfwjNIYD1IGBhW0zzCoM0eZ+tdnUy7wUD1nWOXtPtB63tr4Yek9d9poVlVGmidAs6Y8JYxW9Bj3kaJgmHPEJwpTUXTBi3e7cR224JC17YeXV20PNAQ2oBMSuZDxBkYEE0hVVeVHagiZvu6op24VxoZYt/qQi1CDCE8kZQf2edYhwANDmRUFFhTtaYojjEihgQj1yBaG+pNmK8vv+A+1hn+oHszbKhEBaonliYPctqt/YS12YJj7cXv6v2g/8OzO54xBt0+0cQouyAHfZbbOckOgU1yt5fNGFyvF+JvAAB01d3Hiwtwk8sMBrSJPATkOcRaWdHBzS6fXyOCmM= ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T21:55:02.302 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ppzNzLTpeplBS4fNfSgIWijhpJVnSscHATdj7ONvLCoKNYlNoGzDnJRht9erX01bWiOH4Z/bWqFOS2RAq1d50EE1N2NYCF3NA4i9iWR5J6Wnj8tdiRecO9/j/am+MUl0URPMiKUjqlSk8nxgD3CmJkouv7+em8Mv9eVVZwGQqiYXoNFVfwjNIYD1IGBhW0zzCoM0eZ+tdnUy7wUD1nWOXtPtB63tr4Yek9d9poVlVGmidAs6Y8JYxW9Bj3kaJgmHPEJwpTUXTBi3e7cR224JC17YeXV20PNAQ2oBMSuZDxBkYEE0hVVeVHagiZvu6op24VxoZYt/qQi1CDCE8kZQf2edYhwANDmRUFFhTtaYojjEihgQj1yBaG+pNmK8vv+A+1hn+oHszbKhEBaonliYPctqt/YS12YJj7cXv6v2g/8OzO54xBt0+0cQouyAHfZbbOckOgU1yt5fNGFyvF+JvAAB01d3Hiwtwk8sMBrSJPATkOcRaWdHBzS6fXyOCmM= ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T21:55:02.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T21:55:02.517 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/mon.smithi194/config 2024-03-26T21:55:02.638 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:02 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/981023498' entity='client.admin' 2024-03-26T21:55:04.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/3906165340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 msgr2=0x7fa4bc0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:04.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 --2- 172.21.15.194:0/3906165340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc0ffa40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009910 tx=0x7fa4ac009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:04.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/3906165340 shutdown_connections 2024-03-26T21:55:04.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 --2- 172.21.15.194:0/3906165340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc0ffa40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:04.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/3906165340 >> 172.21.15.194:0/3906165340 conn(0x7fa4bc0fab80 msgr2=0x7fa4bc0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/3906165340 shutdown_connections 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.640+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/3906165340 wait complete. 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c3e4f700 1 Processor -- start 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c3e4f700 1 -- start start 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c3e4f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c3e4f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc198330 con 0x7fa4bc0ff620 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c1beb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c1beb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39792/0 (socket says 172.21.15.194:39792) 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c1beb700 1 -- 172.21.15.194:0/1646176942 learned_addr learned my addr 172.21.15.194:0/1646176942 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.641+0000 7fa4c1beb700 1 -- 172.21.15.194:0/1646176942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0095f0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4c1beb700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc197df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac006010 tx=0x7fa4ac0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ac01d070 con 0x7fa4bc0ff620 2024-03-26T21:55:04.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc198530 con 0x7fa4bc0ff620 2024-03-26T21:55:04.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4ac00be20 con 0x7fa4bc0ff620 2024-03-26T21:55:04.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ac022de0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc1989d0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.642+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa4ac00f4b0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.643+0000 7fa4b2ffd700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4a80387b0 0x7fa4a803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:04.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.643+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa4ac04cab0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.643+0000 7fa4c13ea700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4a80387b0 0x7fa4a803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:04.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.643+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc191f70 con 0x7fa4bc0ff620 2024-03-26T21:55:04.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.643+0000 7fa4c13ea700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4a80387b0 0x7fa4a803ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:04.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.648+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa4ac027030 con 0x7fa4bc0ff620 2024-03-26T21:55:04.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.848+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa4bc0623e0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.851+0000 7fa4b2ffd700 1 -- 172.21.15.194:0/1646176942 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa4ac020ba0 con 0x7fa4bc0ff620 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4a80387b0 msgr2=0x7fa4a803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4a80387b0 0x7fa4a803ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 msgr2=0x7fa4bc197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc197df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac006010 tx=0x7fa4ac0050b0 comp rx=0 tx=0).stop 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 shutdown_connections 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4a80387b0 0x7fa4a803ac70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 --2- 172.21.15.194:0/1646176942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4bc0ff620 0x7fa4bc197df0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:04.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.853+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 >> 172.21.15.194:0/1646176942 conn(0x7fa4bc0fab80 msgr2=0x7fa4bc18f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:04.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.854+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 shutdown_connections 2024-03-26T21:55:04.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:04.854+0000 7fa4c3e4f700 1 -- 172.21.15.194:0/1646176942 wait complete. 2024-03-26T21:55:05.362 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:05.362 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:05.362 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:55:05.362 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:05.362 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi194", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-26T21:55:05.363 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi194", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-26T21:55:05.363 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:55:05.363 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: Deploying daemon ceph-exporter.smithi194 on smithi194 2024-03-26T21:55:05.363 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:05 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/1646176942' entity='client.admin' 2024-03-26T21:55:05.693 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T21:55:05.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T21:55:06.408 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/mon.smithi194/config 2024-03-26T21:55:08.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 -- 172.21.15.194:0/2343053968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 msgr2=0x7f3938100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 --2- 172.21.15.194:0/2343053968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938100b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f392c009910 tx=0x7f392c009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 -- 172.21.15.194:0/2343053968 shutdown_connections 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 --2- 172.21.15.194:0/2343053968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938100b70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 -- 172.21.15.194:0/2343053968 >> 172.21.15.194:0/2343053968 conn(0x7f39380fbcf0 msgr2=0x7f39380fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 -- 172.21.15.194:0/2343053968 shutdown_connections 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.883+0000 7f393f016700 1 -- 172.21.15.194:0/2343053968 wait complete. 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393f016700 1 Processor -- start 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393f016700 1 -- start start 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393f016700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:08.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393f016700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938074f60 con 0x7f3938100750 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393cdb2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393cdb2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39810/0 (socket says 172.21.15.194:39810) 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393cdb2700 1 -- 172.21.15.194:0/4079747414 learned_addr learned my addr 172.21.15.194:0/4079747414 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.884+0000 7f393cdb2700 1 -- 172.21.15.194:0/4079747414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0095f0 con 0x7f3938100750 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.885+0000 7f393cdb2700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938074a20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f392c006010 tx=0x7f392c0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.885+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f392c01d070 con 0x7f3938100750 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.885+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f392c00be20 con 0x7f3938100750 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.885+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39380731a0 con 0x7f3938100750 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.885+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f392c022de0 con 0x7f3938100750 2024-03-26T21:55:08.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.885+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3938073640 con 0x7f3938100750 2024-03-26T21:55:08.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.886+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f392c00f4b0 con 0x7f3938100750 2024-03-26T21:55:08.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.886+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3938106eb0 con 0x7f3938100750 2024-03-26T21:55:08.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.886+0000 7f3935ffb700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f39280387b0 0x7f392803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:08.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.886+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f392c04d8d0 con 0x7f3938100750 2024-03-26T21:55:08.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.886+0000 7f3937fff700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f39280387b0 0x7f392803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:08.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.887+0000 7f3937fff700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f39280387b0 0x7f392803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3924006fd0 tx=0x7f3924006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:08.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:08.892+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f392c020ba0 con 0x7f3938100750 2024-03-26T21:55:09.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.105+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f3938073940 con 0x7f39280387b0 2024-03-26T21:55:09.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.109+0000 7f3935ffb700 1 -- 172.21.15.194:0/4079747414 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3938073940 con 0x7f39280387b0 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f39280387b0 msgr2=0x7f392803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f39280387b0 0x7f392803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3924006fd0 tx=0x7f3924006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 msgr2=0x7f3938074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938074a20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f392c006010 tx=0x7f392c0050b0 comp rx=0 tx=0).stop 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 shutdown_connections 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f39280387b0 0x7f392803ac70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 --2- 172.21.15.194:0/4079747414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3938100750 0x7f3938074a20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 >> 172.21.15.194:0/4079747414 conn(0x7f39380fbcf0 msgr2=0x7f39380fc840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:09.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 shutdown_connections 2024-03-26T21:55:09.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:09.111+0000 7f393f016700 1 -- 172.21.15.194:0/4079747414 wait complete. 2024-03-26T21:55:09.763 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-03-26T21:55:09.764 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:55:09.764 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-03-26T21:55:09.782 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-26T21:55:09.782 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:55:09.839 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-03-26T21:55:09.839 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph orch host add smithi196 2024-03-26T21:55:10.059 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/mon.smithi194/config 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi194", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T21:55:10.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi194", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-26T21:55:10.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:10 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:55:11.381 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:11 smithi194 ceph-mon[29693]: Deploying daemon crash.smithi194 on smithi194 2024-03-26T21:55:11.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 -- 172.21.15.194:0/2587180191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 msgr2=0x7f1984100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:11.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 --2- 172.21.15.194:0/2587180191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984100b40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f197c009910 tx=0x7f197c009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:11.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 -- 172.21.15.194:0/2587180191 shutdown_connections 2024-03-26T21:55:11.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 --2- 172.21.15.194:0/2587180191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984100b40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:11.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 -- 172.21.15.194:0/2587180191 >> 172.21.15.194:0/2587180191 conn(0x7f19840fbcc0 msgr2=0x7f19840fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 -- 172.21.15.194:0/2587180191 shutdown_connections 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.808+0000 7f1989f85700 1 -- 172.21.15.194:0/2587180191 wait complete. 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f1989f85700 1 Processor -- start 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f1989f85700 1 -- start start 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f1989f85700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f1989f85700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1984074f60 con 0x7f1984100720 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f19837fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f19837fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39846/0 (socket says 172.21.15.194:39846) 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f19837fe700 1 -- 172.21.15.194:0/3190095684 learned_addr learned my addr 172.21.15.194:0/3190095684 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.809+0000 7f19837fe700 1 -- 172.21.15.194:0/3190095684 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c0095f0 con 0x7f1984100720 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.810+0000 7f19837fe700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984074a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f197c006010 tx=0x7f197c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.810+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f197c01d070 con 0x7f1984100720 2024-03-26T21:55:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.810+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19840731a0 con 0x7f1984100720 2024-03-26T21:55:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.810+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f197c005dc0 con 0x7f1984100720 2024-03-26T21:55:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.810+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f197c022e90 con 0x7f1984100720 2024-03-26T21:55:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.810+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1984073640 con 0x7f1984100720 2024-03-26T21:55:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.811+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f197c00f4f0 con 0x7f1984100720 2024-03-26T21:55:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.811+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1984106e80 con 0x7f1984100720 2024-03-26T21:55:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.811+0000 7f1980ff9700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f196c038760 0x7f196c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.811+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f197c04cdf0 con 0x7f1984100720 2024-03-26T21:55:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.811+0000 7f1982ffd700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f196c038760 0x7f196c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.811+0000 7f1982ffd700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f196c038760 0x7f196c03ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f197800ad30 tx=0x7f19780093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:11.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:11.816+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f197c02a950 con 0x7f1984100720 2024-03-26T21:55:12.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:12.018+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}) v1 -- 0x7f1984073880 con 0x7f196c038760 2024-03-26T21:55:13.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:13 smithi194 ceph-mon[29693]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:55:13.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:13 smithi194 ceph-mon[29693]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:13.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:13 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:13.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:13 smithi194 ceph-mon[29693]: Deploying cephadm binary to smithi196 2024-03-26T21:55:14.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.769+0000 7f1980ff9700 1 -- 172.21.15.194:0/3190095684 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1984073880 con 0x7f196c038760 2024-03-26T21:55:14.767 INFO:teuthology.orchestra.run.smithi194.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f196c038760 msgr2=0x7f196c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f196c038760 0x7f196c03ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f197800ad30 tx=0x7f19780093f0 comp rx=0 tx=0).stop 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 msgr2=0x7f1984074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984074a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f197c006010 tx=0x7f197c009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 shutdown_connections 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f196c038760 0x7f196c03ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 --2- 172.21.15.194:0/3190095684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1984100720 0x7f1984074a20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 >> 172.21.15.194:0/3190095684 conn(0x7f19840fbcc0 msgr2=0x7f19840fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 shutdown_connections 2024-03-26T21:55:14.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:14.774+0000 7f1989f85700 1 -- 172.21.15.194:0/3190095684 wait complete. 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: Deploying daemon node-exporter.smithi194 on smithi194 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:15.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:15.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:15 smithi194 ceph-mon[29693]: Added host smithi196 2024-03-26T21:55:15.407 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T21:55:15.634 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/mon.smithi194/config 2024-03-26T21:55:17.897 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:17 smithi194 ceph-mon[29693]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:17.897 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:17 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:17.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 -- 172.21.15.194:0/1036887168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 msgr2=0x7f1640100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:17.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 --2- 172.21.15.194:0/1036887168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640100b60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1634009910 tx=0x7f1634009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:17.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 -- 172.21.15.194:0/1036887168 shutdown_connections 2024-03-26T21:55:17.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 --2- 172.21.15.194:0/1036887168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640100b60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:17.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 -- 172.21.15.194:0/1036887168 >> 172.21.15.194:0/1036887168 conn(0x7f16400fbd00 msgr2=0x7f16400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:17.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 -- 172.21.15.194:0/1036887168 shutdown_connections 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.892+0000 7f1646dd8700 1 -- 172.21.15.194:0/1036887168 wait complete. 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1646dd8700 1 Processor -- start 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1646dd8700 1 -- start start 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1646dd8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1646dd8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16401960d0 con 0x7f1640100740 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1644b74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1644b74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:37492/0 (socket says 172.21.15.194:37492) 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.893+0000 7f1644b74700 1 -- 172.21.15.194:0/3985917873 learned_addr learned my addr 172.21.15.194:0/3985917873 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f1644b74700 1 -- 172.21.15.194:0/3985917873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340095f0 con 0x7f1640100740 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f1644b74700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640195b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1634006010 tx=0x7f1634009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f163401d070 con 0x7f1640100740 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16401962d0 con 0x7f1640100740 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1634005dc0 con 0x7f1640100740 2024-03-26T21:55:17.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1634022e90 con 0x7f1640100740 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.894+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1640196770 con 0x7f1640100740 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.895+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f163400f4f0 con 0x7f1640100740 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.895+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164018f570 con 0x7f1640100740 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.895+0000 7f163dffb700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1630038760 0x7f163003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.895+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f163404ccb0 con 0x7f1640100740 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.895+0000 7f163ffff700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1630038760 0x7f163003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.895+0000 7f163ffff700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1630038760 0x7f163003ac20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f162c006fd0 tx=0x7f162c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:17.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:17.900+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f163402a950 con 0x7f1640100740 2024-03-26T21:55:18.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.099+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16400611f0 con 0x7f1630038760 2024-03-26T21:55:18.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.101+0000 7f163dffb700 1 -- 172.21.15.194:0/3985917873 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f16400611f0 con 0x7f1630038760 2024-03-26T21:55:18.097 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-26T21:55:18.097 INFO:teuthology.orchestra.run.smithi194.stdout:[{"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-03-26T21:55:18.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1630038760 msgr2=0x7f163003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1630038760 0x7f163003ac20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f162c006fd0 tx=0x7f162c006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 msgr2=0x7f1640195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640195b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1634006010 tx=0x7f1634009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 shutdown_connections 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1630038760 0x7f163003ac20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 --2- 172.21.15.194:0/3985917873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1640100740 0x7f1640195b90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 >> 172.21.15.194:0/3985917873 conn(0x7f16400fbd00 msgr2=0x7f16400fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 shutdown_connections 2024-03-26T21:55:18.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:18.103+0000 7f1646dd8700 1 -- 172.21.15.194:0/3985917873 wait complete. 2024-03-26T21:55:18.963 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T21:55:18.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T21:55:19.608 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/mon.smithi194/config 2024-03-26T21:55:19.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:19 smithi194 ceph-mon[29693]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:55:19.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:19 smithi194 ceph-mon[29693]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:19.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:19 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:19.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:19 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:19.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:19 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:19.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:19 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:20.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:20 smithi194 ceph-mon[29693]: Deploying daemon alertmanager.smithi194 on smithi194 2024-03-26T21:55:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 -- 172.21.15.194:0/4015058981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 msgr2=0x7f9df0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 --2- 172.21.15.194:0/4015058981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0100b40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009910 tx=0x7f9de0009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 -- 172.21.15.194:0/4015058981 shutdown_connections 2024-03-26T21:55:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 --2- 172.21.15.194:0/4015058981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0100b40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 -- 172.21.15.194:0/4015058981 >> 172.21.15.194:0/4015058981 conn(0x7f9df00fbcc0 msgr2=0x7f9df00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 -- 172.21.15.194:0/4015058981 shutdown_connections 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.487+0000 7f9df5c6a700 1 -- 172.21.15.194:0/4015058981 wait complete. 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9df5c6a700 1 Processor -- start 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9df5c6a700 1 -- start start 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9df5c6a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9df5c6a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df0196100 con 0x7f9df0100720 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9def7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9def7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:37508/0 (socket says 172.21.15.194:37508) 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.488+0000 7f9def7fe700 1 -- 172.21.15.194:0/3610627690 learned_addr learned my addr 172.21.15.194:0/3610627690 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9def7fe700 1 -- 172.21.15.194:0/3610627690 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00095f0 con 0x7f9df0100720 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9def7fe700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0195bc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006010 tx=0x7f9de00050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9de001d070 con 0x7f9df0100720 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df0196300 con 0x7f9df0100720 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9de000be20 con 0x7f9df0100720 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9de0022de0 con 0x7f9df0100720 2024-03-26T21:55:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.489+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df01967a0 con 0x7f9df0100720 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.490+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9de000b960 con 0x7f9df0100720 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.490+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df018f550 con 0x7f9df0100720 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.490+0000 7f9decff9700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dd80387b0 0x7f9dd803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.490+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9de004ca10 con 0x7f9df0100720 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.490+0000 7f9deeffd700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dd80387b0 0x7f9dd803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.491+0000 7f9deeffd700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dd80387b0 0x7f9dd803ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9de4006fd0 tx=0x7f9de4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.495+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9de002bd00 con 0x7f9df0100720 2024-03-26T21:55:21.700 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:21.704+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9df002ccb0 con 0x7f9df0100720 2024-03-26T21:55:22.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:21 smithi194 ceph-mon[29693]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:22.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.680+0000 7f9decff9700 1 -- 172.21.15.194:0/3610627690 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9de002a360 con 0x7f9df0100720 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dd80387b0 msgr2=0x7f9dd803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dd80387b0 0x7f9dd803ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9de4006fd0 tx=0x7f9de4006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 msgr2=0x7f9df0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0195bc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006010 tx=0x7f9de00050b0 comp rx=0 tx=0).stop 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 shutdown_connections 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dd80387b0 0x7f9dd803ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 --2- 172.21.15.194:0/3610627690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9df0100720 0x7f9df0195bc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 >> 172.21.15.194:0/3610627690 conn(0x7f9df00fbcc0 msgr2=0x7f9df00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 shutdown_connections 2024-03-26T21:55:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-26T21:55:22.682+0000 7f9df5c6a700 1 -- 172.21.15.194:0/3610627690 wait complete. 2024-03-26T21:55:22.680 INFO:teuthology.orchestra.run.smithi194.stderr:adjusted tunables profile to default 2024-03-26T21:55:23.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:22 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/3610627690' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:55:23.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:22 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:23.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:23 smithi194 ceph-mon[29693]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:23.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:23 smithi194 ceph-mon[29693]: from='client.? 172.21.15.194:0/3610627690' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:55:23.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:23 smithi194 ceph-mon[29693]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:55:24.139 INFO:tasks.cephadm:Adding mon.smithi194 on smithi194 2024-03-26T21:55:24.139 INFO:tasks.cephadm:Adding mon.smithi196 on smithi196 2024-03-26T21:55:24.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph orch apply mon '2;smithi194:172.21.15.194=smithi194;smithi196:172.21.15.196=smithi196' 2024-03-26T21:55:24.342 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:24.390 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:25.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:25 smithi194 ceph-mon[29693]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:25.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 -- 172.21.15.196:0/3816776592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 msgr2=0x7f61040fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:25.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 --2- 172.21.15.196:0/3816776592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f61040fee10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6100009910 tx=0x7f6100009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:25.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 -- 172.21.15.196:0/3816776592 shutdown_connections 2024-03-26T21:55:25.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 --2- 172.21.15.196:0/3816776592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f61040fee10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:25.891 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 -- 172.21.15.196:0/3816776592 >> 172.21.15.196:0/3816776592 conn(0x7f61040fa370 msgr2=0x7f61040fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:25.892 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 -- 172.21.15.196:0/3816776592 shutdown_connections 2024-03-26T21:55:25.892 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.894+0000 7f610bd80700 1 -- 172.21.15.196:0/3816776592 wait complete. 2024-03-26T21:55:25.892 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.895+0000 7f610bd80700 1 Processor -- start 2024-03-26T21:55:25.892 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.895+0000 7f610bd80700 1 -- start start 2024-03-26T21:55:25.892 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.895+0000 7f610bd80700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f6104101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:25.892 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.895+0000 7f610bd80700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6104101740 con 0x7f61040fe9f0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.895+0000 7f6109b1c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f6104101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f6109b1c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f6104101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:33706/0 (socket says 172.21.15.196:33706) 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f6109b1c700 1 -- 172.21.15.196:0/4034766423 learned_addr learned my addr 172.21.15.196:0/4034766423 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f6109b1c700 1 -- 172.21.15.196:0/4034766423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000095f0 con 0x7f61040fe9f0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f6109b1c700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f6104101200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6100006010 tx=0x7f6100009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f610001d070 con 0x7f61040fe9f0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6100005dc0 con 0x7f61040fe9f0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61040ff980 con 0x7f61040fe9f0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6100022e90 con 0x7f61040fe9f0 2024-03-26T21:55:25.894 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.896+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61040ffe20 con 0x7f61040fe9f0 2024-03-26T21:55:25.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.897+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6100022470 con 0x7f61040fe9f0 2024-03-26T21:55:25.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.897+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6104106ea0 con 0x7f61040fe9f0 2024-03-26T21:55:25.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.897+0000 7f60faffd700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f60f0038760 0x7f60f003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:25.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.897+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f610004c670 con 0x7f61040fe9f0 2024-03-26T21:55:25.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.898+0000 7f610931b700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f60f0038760 0x7f60f003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:25.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.898+0000 7f610931b700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f60f0038760 0x7f60f003ac20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f60f4006fd0 tx=0x7f60f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:25.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:25.903+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f610002a360 con 0x7f61040fe9f0 2024-03-26T21:55:26.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.106+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 --> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi194:172.21.15.194=smithi194;smithi196:172.21.15.196=smithi196", "target": ["mon-mgr", ""]}) v1 -- 0x7f61040ff520 con 0x7f60f0038760 2024-03-26T21:55:26.109 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.113+0000 7f60faffd700 1 -- 172.21.15.196:0/4034766423 <== mgr.14162 v2:172.21.15.194:6800/3128608440 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f61040ff520 con 0x7f60f0038760 2024-03-26T21:55:26.109 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.114+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f60f0038760 msgr2=0x7f60f003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.114+0000 7f610bd80700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f60f0038760 0x7f60f003ac20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f60f4006fd0 tx=0x7f60f4006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.114+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 msgr2=0x7f6104101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.114+0000 7f610bd80700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f6104101200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6100006010 tx=0x7f6100009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.115+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 shutdown_connections 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.115+0000 7f610bd80700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f60f0038760 0x7f60f003ac20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.115+0000 7f610bd80700 1 --2- 172.21.15.196:0/4034766423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61040fe9f0 0x7f6104101200 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.115+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 >> 172.21.15.196:0/4034766423 conn(0x7f61040fa370 msgr2=0x7f61040fae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.115+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 shutdown_connections 2024-03-26T21:55:26.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:26.115+0000 7f610bd80700 1 -- 172.21.15.196:0/4034766423 wait complete. 2024-03-26T21:55:26.622 DEBUG:teuthology.orchestra.run.smithi196:mon.smithi196> sudo journalctl -f -n 0 -u ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196.service 2024-03-26T21:55:26.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:26.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:26.850 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:26.895 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:27.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:27 smithi194 ceph-mon[29693]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi194:172.21.15.194=smithi194;smithi196:172.21.15.196=smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:55:27.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:27 smithi194 ceph-mon[29693]: Saving service mon spec with placement smithi194:172.21.15.194=smithi194;smithi196:172.21.15.196=smithi196;count:2 2024-03-26T21:55:27.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:27 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:27.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:27 smithi194 ceph-mon[29693]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:28.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 -- 172.21.15.196:0/966593777 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 msgr2=0x7f570c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:28.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 --2- 172.21.15.196:0/966593777 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0ffa40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009910 tx=0x7f56fc009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:28.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 -- 172.21.15.196:0/966593777 shutdown_connections 2024-03-26T21:55:28.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 --2- 172.21.15.196:0/966593777 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0ffa40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:28.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 -- 172.21.15.196:0/966593777 >> 172.21.15.196:0/966593777 conn(0x7f570c0fab80 msgr2=0x7f570c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:28.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 -- 172.21.15.196:0/966593777 shutdown_connections 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.295+0000 7f5714534700 1 -- 172.21.15.196:0/966593777 wait complete. 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f5714534700 1 Processor -- start 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f5714534700 1 -- start start 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f5714534700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f5714534700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c068e10 con 0x7f570c0ff620 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f57122d0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f57122d0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0688d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:33724/0 (socket says 172.21.15.196:33724) 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f57122d0700 1 -- 172.21.15.196:0/3175977797 learned_addr learned my addr 172.21.15.196:0/3175977797 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.296+0000 7f57122d0700 1 -- 172.21.15.196:0/3175977797 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0095f0 con 0x7f570c0ff620 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.297+0000 7f57122d0700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0688d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f56fc006010 tx=0x7f56fc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.297+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56fc01d070 con 0x7f570c0ff620 2024-03-26T21:55:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.297+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56fc005dc0 con 0x7f570c0ff620 2024-03-26T21:55:28.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.297+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56fc022e90 con 0x7f570c0ff620 2024-03-26T21:55:28.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.297+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c196700 con 0x7f570c0ff620 2024-03-26T21:55:28.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.297+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c19af60 con 0x7f570c0ff620 2024-03-26T21:55:28.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.298+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c18e450 con 0x7f570c0ff620 2024-03-26T21:55:28.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.298+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f56fc022470 con 0x7f570c0ff620 2024-03-26T21:55:28.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.298+0000 7f57037fe700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56f8038760 0x7f56f803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:28.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.299+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f56fc04c540 con 0x7f570c0ff620 2024-03-26T21:55:28.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.299+0000 7f5711acf700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56f8038760 0x7f56f803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:28.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.299+0000 7f5711acf700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56f8038760 0x7f56f803ac20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5708006fd0 tx=0x7f5708006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:28.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.304+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f56fc00f460 con 0x7f570c0ff620 2024-03-26T21:55:28.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.572+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f570c0623e0 con 0x7f570c0ff620 2024-03-26T21:55:28.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.572+0000 7f57037fe700 1 -- 172.21.15.196:0/3175977797 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f56fc027030 con 0x7f570c0ff620 2024-03-26T21:55:28.569 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:28.569 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56f8038760 msgr2=0x7f56f803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56f8038760 0x7f56f803ac20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5708006fd0 tx=0x7f5708006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 msgr2=0x7f570c0688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0688d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f56fc006010 tx=0x7f56fc009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 shutdown_connections 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f56f8038760 0x7f56f803ac20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 --2- 172.21.15.196:0/3175977797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f570c0ff620 0x7f570c0688d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 >> 172.21.15.196:0/3175977797 conn(0x7f570c0fab80 msgr2=0x7f570c0fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 shutdown_connections 2024-03-26T21:55:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:28.575+0000 7f5714534700 1 -- 172.21.15.196:0/3175977797 wait complete. 2024-03-26T21:55:28.572 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3175977797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:55:29.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:55:29.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:29.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:29.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:29 smithi194 ceph-mon[29693]: Deploying daemon grafana.smithi194 on smithi194 2024-03-26T21:55:30.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:30.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:30.361 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:30.407 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:31.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- 172.21.15.196:0/725853054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 msgr2=0x7f2e9c102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 --2- 172.21.15.196:0/725853054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c102160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009910 tx=0x7f2e98009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- 172.21.15.196:0/725853054 shutdown_connections 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 --2- 172.21.15.196:0/725853054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c102160 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- 172.21.15.196:0/725853054 >> 172.21.15.196:0/725853054 conn(0x7f2e9c0fa380 msgr2=0x7f2e9c0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- 172.21.15.196:0/725853054 shutdown_connections 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- 172.21.15.196:0/725853054 wait complete. 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 Processor -- start 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- start start 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.839+0000 7f2ea4735700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c196140 con 0x7f2e9c101d40 2024-03-26T21:55:31.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea24d1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea24d1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:33736/0 (socket says 172.21.15.196:33736) 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea24d1700 1 -- 172.21.15.196:0/2100912308 learned_addr learned my addr 172.21.15.196:0/2100912308 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea24d1700 1 -- 172.21.15.196:0/2100912308 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980095f0 con 0x7f2e9c101d40 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea24d1700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c195c00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e98006010 tx=0x7f2e98009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e9801d070 con 0x7f2e9c101d40 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e98005dc0 con 0x7f2e9c101d40 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9c196340 con 0x7f2e9c101d40 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e98022e90 con 0x7f2e9c101d40 2024-03-26T21:55:31.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.840+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9c1967e0 con 0x7f2e9c101d40 2024-03-26T21:55:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.841+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2e9800f4f0 con 0x7f2e9c101d40 2024-03-26T21:55:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.841+0000 7f2e937fe700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2e88038760 0x7f2e8803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.841+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e9804c8b0 con 0x7f2e9c101d40 2024-03-26T21:55:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.841+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9c18f590 con 0x7f2e9c101d40 2024-03-26T21:55:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.841+0000 7f2ea1cd0700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2e88038760 0x7f2e8803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.842+0000 7f2ea1cd0700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2e88038760 0x7f2e8803ac20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c006fd0 tx=0x7f2e8c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:31.846 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:31.850+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e98027070 con 0x7f2e9c101d40 2024-03-26T21:55:31.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:31 smithi194 ceph-mon[29693]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:32.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.126+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e9c0623e0 con 0x7f2e9c101d40 2024-03-26T21:55:32.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.127+0000 7f2e937fe700 1 -- 172.21.15.196:0/2100912308 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2e9802a720 con 0x7f2e9c101d40 2024-03-26T21:55:32.124 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:32.124 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:32.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2e88038760 msgr2=0x7f2e8803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:32.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2e88038760 0x7f2e8803ac20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c006fd0 tx=0x7f2e8c006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:32.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 msgr2=0x7f2e9c195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:32.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c195c00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e98006010 tx=0x7f2e98009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 shutdown_connections 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2e88038760 0x7f2e8803ac20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 --2- 172.21.15.196:0/2100912308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e9c101d40 0x7f2e9c195c00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 >> 172.21.15.196:0/2100912308 conn(0x7f2e9c0fa380 msgr2=0x7f2e9c0faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 shutdown_connections 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:32.129+0000 7f2ea4735700 1 -- 172.21.15.196:0/2100912308 wait complete. 2024-03-26T21:55:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:32.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:32 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2100912308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:33.639 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:33.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:33.834 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:33.880 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:34.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:33 smithi194 ceph-mon[29693]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:34.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:33 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:35.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 -- 172.21.15.196:0/3585698911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 msgr2=0x7f620c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:35.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 --2- 172.21.15.196:0/3585698911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c0ffa40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009910 tx=0x7f61fc009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 -- 172.21.15.196:0/3585698911 shutdown_connections 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 --2- 172.21.15.196:0/3585698911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c0ffa40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 -- 172.21.15.196:0/3585698911 >> 172.21.15.196:0/3585698911 conn(0x7f620c0fabc0 msgr2=0x7f620c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 -- 172.21.15.196:0/3585698911 shutdown_connections 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.236+0000 7f62146b4700 1 -- 172.21.15.196:0/3585698911 wait complete. 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f62146b4700 1 Processor -- start 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f62146b4700 1 -- start start 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f62146b4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f62146b4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c196110 con 0x7f620c0ff620 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f6212450700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f6212450700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45312/0 (socket says 172.21.15.196:45312) 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.237+0000 7f6212450700 1 -- 172.21.15.196:0/3461930008 learned_addr learned my addr 172.21.15.196:0/3461930008 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f6212450700 1 -- 172.21.15.196:0/3461930008 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0095f0 con 0x7f620c0ff620 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f6212450700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c195bd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f61fc006010 tx=0x7f61fc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:35.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61fc01d070 con 0x7f620c0ff620 2024-03-26T21:55:35.236 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61fc005dc0 con 0x7f620c0ff620 2024-03-26T21:55:35.236 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c196310 con 0x7f620c0ff620 2024-03-26T21:55:35.236 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61fc022e90 con 0x7f620c0ff620 2024-03-26T21:55:35.236 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.238+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c1967b0 con 0x7f620c0ff620 2024-03-26T21:55:35.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.239+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f61fc00b960 con 0x7f620c0ff620 2024-03-26T21:55:35.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.239+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c0623e0 con 0x7f620c0ff620 2024-03-26T21:55:35.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.239+0000 7f62037fe700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f61f8038760 0x7f61f803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:35.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.239+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f61fc04c880 con 0x7f620c0ff620 2024-03-26T21:55:35.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.239+0000 7f6211c4f700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f61f8038760 0x7f61f803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:35.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.240+0000 7f6211c4f700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f61f8038760 0x7f61f803ac20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6208006fd0 tx=0x7f6208006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:35.240 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.244+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61fc022750 con 0x7f620c0ff620 2024-03-26T21:55:35.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:34 smithi194 ceph-mon[29693]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:35.505 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.509+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f620c105650 con 0x7f620c0ff620 2024-03-26T21:55:35.506 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.510+0000 7f62037fe700 1 -- 172.21.15.196:0/3461930008 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f61fc027020 con 0x7f620c0ff620 2024-03-26T21:55:35.506 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:35.506 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f61f8038760 msgr2=0x7f61f803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f61f8038760 0x7f61f803ac20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6208006fd0 tx=0x7f6208006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 msgr2=0x7f620c195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c195bd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f61fc006010 tx=0x7f61fc009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 shutdown_connections 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f61f8038760 0x7f61f803ac20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 --2- 172.21.15.196:0/3461930008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f620c0ff620 0x7f620c195bd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:35.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 >> 172.21.15.196:0/3461930008 conn(0x7f620c0fabc0 msgr2=0x7f620c068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:35.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 shutdown_connections 2024-03-26T21:55:35.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:35.512+0000 7f62146b4700 1 -- 172.21.15.196:0/3461930008 wait complete. 2024-03-26T21:55:35.509 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:36.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:35 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3461930008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:37.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:37.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:37.290 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:37.336 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:37.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:36 smithi194 ceph-mon[29693]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:38.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 -- 172.21.15.196:0/2285739456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 msgr2=0x7f64fc0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:38.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 --2- 172.21.15.196:0/2285739456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc0ffa40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009910 tx=0x7f64ec009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 -- 172.21.15.196:0/2285739456 shutdown_connections 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 --2- 172.21.15.196:0/2285739456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc0ffa40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 -- 172.21.15.196:0/2285739456 >> 172.21.15.196:0/2285739456 conn(0x7f64fc0fab80 msgr2=0x7f64fc0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 -- 172.21.15.196:0/2285739456 shutdown_connections 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.754+0000 7f65036f2700 1 -- 172.21.15.196:0/2285739456 wait complete. 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f65036f2700 1 Processor -- start 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f65036f2700 1 -- start start 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f65036f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc18c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f65036f2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc18cbd0 con 0x7f64fc0ff620 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f650148e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc18c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:38.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f650148e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc18c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45330/0 (socket says 172.21.15.196:45330) 2024-03-26T21:55:38.753 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.755+0000 7f650148e700 1 -- 172.21.15.196:0/3898848307 learned_addr learned my addr 172.21.15.196:0/3898848307 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:38.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f650148e700 1 -- 172.21.15.196:0/3898848307 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec0095f0 con 0x7f64fc0ff620 2024-03-26T21:55:38.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f650148e700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc18c690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f64ec006010 tx=0x7f64ec0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:38.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64ec01d070 con 0x7f64fc0ff620 2024-03-26T21:55:38.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64ec00be20 con 0x7f64fc0ff620 2024-03-26T21:55:38.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc18ae60 con 0x7f64fc0ff620 2024-03-26T21:55:38.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64ec022de0 con 0x7f64fc0ff620 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.756+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc18b300 con 0x7f64fc0ff620 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.757+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f64ec00b960 con 0x7f64fc0ff620 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.757+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc105d80 con 0x7f64fc0ff620 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.757+0000 7f64f27fc700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64e80387b0 0x7f64e803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.757+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f64ec04c980 con 0x7f64fc0ff620 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.757+0000 7f6500c8d700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64e80387b0 0x7f64e803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:38.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.758+0000 7f6500c8d700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64e80387b0 0x7f64e803ac70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64f8006fd0 tx=0x7f64f8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:38.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:38.762+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f64ec0226c0 con 0x7f64fc0ff620 2024-03-26T21:55:39.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.031+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64fc0623e0 con 0x7f64fc0ff620 2024-03-26T21:55:39.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.032+0000 7f64f27fc700 1 -- 172.21.15.196:0/3898848307 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f64ec027030 con 0x7f64fc0ff620 2024-03-26T21:55:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:39.028 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64e80387b0 msgr2=0x7f64e803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64e80387b0 0x7f64e803ac70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64f8006fd0 tx=0x7f64f8006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 msgr2=0x7f64fc18c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc18c690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f64ec006010 tx=0x7f64ec0050b0 comp rx=0 tx=0).stop 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 shutdown_connections 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64e80387b0 0x7f64e803ac70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 --2- 172.21.15.196:0/3898848307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64fc0ff620 0x7f64fc18c690 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:39.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 >> 172.21.15.196:0/3898848307 conn(0x7f64fc0fab80 msgr2=0x7f64fc0fb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:39.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 shutdown_connections 2024-03-26T21:55:39.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:39.034+0000 7f65036f2700 1 -- 172.21.15.196:0/3898848307 wait complete. 2024-03-26T21:55:39.032 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:39.810 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:39 smithi194 ceph-mon[29693]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:39.811 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:39 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3898848307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:40.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:40.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:40.661 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:40.706 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:41.743 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:41 smithi194 ceph-mon[29693]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:42.091 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.093+0000 7f4288cd7700 1 -- 172.21.15.196:0/1446522432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 msgr2=0x7f4284100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.093+0000 7f4288cd7700 1 --2- 172.21.15.196:0/1446522432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284100b60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4274009910 tx=0x7f4274009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 -- 172.21.15.196:0/1446522432 shutdown_connections 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 --2- 172.21.15.196:0/1446522432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284100b60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 -- 172.21.15.196:0/1446522432 >> 172.21.15.196:0/1446522432 conn(0x7f42840fbd00 msgr2=0x7f42840fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 -- 172.21.15.196:0/1446522432 shutdown_connections 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 -- 172.21.15.196:0/1446522432 wait complete. 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 Processor -- start 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 -- start start 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.094+0000 7f4288cd7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841960d0 con 0x7f4284100740 2024-03-26T21:55:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f428259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f428259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45346/0 (socket says 172.21.15.196:45346) 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f428259c700 1 -- 172.21.15.196:0/2624228772 learned_addr learned my addr 172.21.15.196:0/2624228772 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f428259c700 1 -- 172.21.15.196:0/2624228772 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42740095f0 con 0x7f4284100740 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f428259c700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284195b90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4274006010 tx=0x7f4274009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f427401d070 con 0x7f4284100740 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4274005dc0 con 0x7f4284100740 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42841962d0 con 0x7f4284100740 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274022e90 con 0x7f4284100740 2024-03-26T21:55:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.095+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4284196770 con 0x7f4284100740 2024-03-26T21:55:42.094 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.096+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f427400f4b0 con 0x7f4284100740 2024-03-26T21:55:42.094 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.096+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428418f570 con 0x7f4284100740 2024-03-26T21:55:42.094 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.096+0000 7f42737fe700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f426c038810 0x7f426c03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:42.094 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.096+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f427404ca90 con 0x7f4284100740 2024-03-26T21:55:42.094 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.096+0000 7f4281d9b700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f426c038810 0x7f426c03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:42.094 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.097+0000 7f4281d9b700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f426c038810 0x7f426c03acd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4278006fd0 tx=0x7f4278006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:42.097 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.101+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f427400b960 con 0x7f4284100740 2024-03-26T21:55:42.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.371+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42840623e0 con 0x7f4284100740 2024-03-26T21:55:42.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.371+0000 7f42737fe700 1 -- 172.21.15.196:0/2624228772 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f42840623e0 con 0x7f4284100740 2024-03-26T21:55:42.368 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:42.368 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.373+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f426c038810 msgr2=0x7f426c03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.373+0000 7f4288cd7700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f426c038810 0x7f426c03acd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4278006fd0 tx=0x7f4278006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.373+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 msgr2=0x7f4284195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.373+0000 7f4288cd7700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284195b90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4274006010 tx=0x7f4274009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.374+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 shutdown_connections 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.374+0000 7f4288cd7700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f426c038810 0x7f426c03acd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:42.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.374+0000 7f4288cd7700 1 --2- 172.21.15.196:0/2624228772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4284100740 0x7f4284195b90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:42.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.374+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 >> 172.21.15.196:0/2624228772 conn(0x7f42840fbd00 msgr2=0x7f42840fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:42.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.374+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 shutdown_connections 2024-03-26T21:55:42.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:42.374+0000 7f4288cd7700 1 -- 172.21.15.196:0/2624228772 wait complete. 2024-03-26T21:55:42.371 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:42.817 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:42 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2624228772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:42.817 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:42 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:42.817 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:42 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:42.817 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:42 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:42.817 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:42 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:42.817 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:42 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:43.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:43.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:43.972 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:43 smithi194 ceph-mon[29693]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:43.972 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:43 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:43.973 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:43 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:43.973 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:43 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:43.973 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:43 smithi194 ceph-mon[29693]: from='mgr.14162 172.21.15.194:0/3140127418' entity='mgr.smithi194.wndbij' 2024-03-26T21:55:44.150 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:44.197 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:44.938 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:44 smithi194 ceph-mon[29693]: Deploying daemon prometheus.smithi194 on smithi194 2024-03-26T21:55:45.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 -- 172.21.15.196:0/1827192451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 msgr2=0x7fb3680fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 --2- 172.21.15.196:0/1827192451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb3680fee30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009910 tx=0x7fb35c009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 -- 172.21.15.196:0/1827192451 shutdown_connections 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 --2- 172.21.15.196:0/1827192451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb3680fee30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 -- 172.21.15.196:0/1827192451 >> 172.21.15.196:0/1827192451 conn(0x7fb3680fa370 msgr2=0x7fb3680fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 -- 172.21.15.196:0/1827192451 shutdown_connections 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.604+0000 7fb36d0e6700 1 -- 172.21.15.196:0/1827192451 wait complete. 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.605+0000 7fb36d0e6700 1 Processor -- start 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.605+0000 7fb36d0e6700 1 -- start start 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.605+0000 7fb36d0e6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb368195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:45.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.605+0000 7fb36d0e6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368196120 con 0x7fb3680fea10 2024-03-26T21:55:45.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb366d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb368195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:45.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb366d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb368195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55352/0 (socket says 172.21.15.196:55352) 2024-03-26T21:55:45.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb366d9d700 1 -- 172.21.15.196:0/3176559457 learned_addr learned my addr 172.21.15.196:0/3176559457 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:45.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb366d9d700 1 -- 172.21.15.196:0/3176559457 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0095f0 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb366d9d700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb368195be0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb35c006010 tx=0x7fb35c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb35c01d070 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb368196320 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb35c005dc0 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb35c022e90 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.606+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3681967c0 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.607+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36818f590 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.607+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb35c00f4f0 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.607+0000 7fb34ffff700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb350038760 0x7fb35003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.607+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb35c04c950 con 0x7fb3680fea10 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.608+0000 7fb36659c700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb350038760 0x7fb35003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:45.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.608+0000 7fb36659c700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb350038760 0x7fb35003ac20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb358006fd0 tx=0x7fb358006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:45.609 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.613+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb35c017440 con 0x7fb3680fea10 2024-03-26T21:55:45.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.889+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3680ff540 con 0x7fb3680fea10 2024-03-26T21:55:45.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.890+0000 7fb34ffff700 1 -- 172.21.15.196:0/3176559457 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb3680ff540 con 0x7fb3680fea10 2024-03-26T21:55:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:45.887 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb350038760 msgr2=0x7fb35003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb350038760 0x7fb35003ac20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb358006fd0 tx=0x7fb358006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 msgr2=0x7fb368195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb368195be0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb35c006010 tx=0x7fb35c009f80 comp rx=0 tx=0).stop 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 shutdown_connections 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb350038760 0x7fb35003ac20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 --2- 172.21.15.196:0/3176559457 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3680fea10 0x7fb368195be0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.892+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 >> 172.21.15.196:0/3176559457 conn(0x7fb3680fa370 msgr2=0x7fb3680fae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.893+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 shutdown_connections 2024-03-26T21:55:45.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:45.893+0000 7fb36d0e6700 1 -- 172.21.15.196:0/3176559457 wait complete. 2024-03-26T21:55:45.890 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:46.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:45 smithi194 ceph-mon[29693]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:47.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:46 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3176559457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:47.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:47.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:47.726 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:47.771 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:48.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:47 smithi194 ceph-mon[29693]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:49.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 -- 172.21.15.196:0/3616218217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 msgr2=0x7f8a500ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 --2- 172.21.15.196:0/3616218217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a500ffa50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009910 tx=0x7f8a40009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 -- 172.21.15.196:0/3616218217 shutdown_connections 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 --2- 172.21.15.196:0/3616218217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a500ffa50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 -- 172.21.15.196:0/3616218217 >> 172.21.15.196:0/3616218217 conn(0x7f8a500fabd0 msgr2=0x7f8a500fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 -- 172.21.15.196:0/3616218217 shutdown_connections 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.268+0000 7f8a55f52700 1 -- 172.21.15.196:0/3616218217 wait complete. 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a55f52700 1 Processor -- start 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a55f52700 1 -- start start 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a55f52700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a50194b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a55f52700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50195070 con 0x7f8a500ff630 2024-03-26T21:55:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a50194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:49.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a50194b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55378/0 (socket says 172.21.15.196:55378) 2024-03-26T21:55:49.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.269+0000 7f8a4f7fe700 1 -- 172.21.15.196:0/1334187815 learned_addr learned my addr 172.21.15.196:0/1334187815 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:49.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a4f7fe700 1 -- 172.21.15.196:0/1334187815 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400095f0 con 0x7f8a500ff630 2024-03-26T21:55:49.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a4f7fe700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a50194b30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006010 tx=0x7f8a40005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a4001d070 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a50195270 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a4000be20 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a40022e40 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.270+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a50195710 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.271+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8a4000b960 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.271+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5018e460 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.271+0000 7f8a4cff9700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8a380387b0 0x7f8a3803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.271+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8a4004c8a0 con 0x7f8a500ff630 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.271+0000 7f8a4effd700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8a380387b0 0x7f8a3803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.272+0000 7f8a4effd700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8a380387b0 0x7f8a3803ac70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8a44006fd0 tx=0x7f8a44006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:49.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.277+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8a40022700 con 0x7f8a500ff630 2024-03-26T21:55:49.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.553+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a500623e0 con 0x7f8a500ff630 2024-03-26T21:55:49.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.554+0000 7f8a4cff9700 1 -- 172.21.15.196:0/1334187815 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8a500623e0 con 0x7f8a500ff630 2024-03-26T21:55:49.550 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:49.550 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.555+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8a380387b0 msgr2=0x7f8a3803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.555+0000 7f8a55f52700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8a380387b0 0x7f8a3803ac70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8a44006fd0 tx=0x7f8a44006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.555+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 msgr2=0x7f8a50194b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.555+0000 7f8a55f52700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a50194b30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006010 tx=0x7f8a40005ee0 comp rx=0 tx=0).stop 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.555+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 shutdown_connections 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.555+0000 7f8a55f52700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8a380387b0 0x7f8a3803ac70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.556+0000 7f8a55f52700 1 --2- 172.21.15.196:0/1334187815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a500ff630 0x7f8a50194b30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.556+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 >> 172.21.15.196:0/1334187815 conn(0x7f8a500fabd0 msgr2=0x7f8a500fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:49.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.556+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 shutdown_connections 2024-03-26T21:55:49.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:49.556+0000 7f8a55f52700 1 -- 172.21.15.196:0/1334187815 wait complete. 2024-03-26T21:55:49.553 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:49.944 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:49 smithi194 ceph-mon[29693]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:49.945 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:49 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1334187815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:51.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:51.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:51.334 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:51.380 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:52.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:51 smithi194 ceph-mon[29693]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:52.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 -- 172.21.15.196:0/748052900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 msgr2=0x7fef900731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:52.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 --2- 172.21.15.196:0/748052900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef900731c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fef80009910 tx=0x7fef80009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 -- 172.21.15.196:0/748052900 shutdown_connections 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 --2- 172.21.15.196:0/748052900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef900731c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 -- 172.21.15.196:0/748052900 >> 172.21.15.196:0/748052900 conn(0x7fef900fbd00 msgr2=0x7fef900fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 -- 172.21.15.196:0/748052900 shutdown_connections 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.728+0000 7fef96a36700 1 -- 172.21.15.196:0/748052900 wait complete. 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.729+0000 7fef96a36700 1 Processor -- start 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.729+0000 7fef96a36700 1 -- start start 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.729+0000 7fef96a36700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef90197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.729+0000 7fef96a36700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901982c0 con 0x7fef90074d60 2024-03-26T21:55:52.728 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.729+0000 7fef8ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef90197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:52.728 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef8ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef90197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:33704/0 (socket says 172.21.15.196:33704) 2024-03-26T21:55:52.728 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef8ffff700 1 -- 172.21.15.196:0/1552590038 learned_addr learned my addr 172.21.15.196:0/1552590038 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:52.728 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef8ffff700 1 -- 172.21.15.196:0/1552590038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef800095f0 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef8ffff700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef90197d80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fef80006010 tx=0x7fef800050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8001d070 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef901984c0 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef90198960 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.730+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef8000be20 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.731+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8000f460 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.731+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fef8000f5c0 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.731+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c005320 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.731+0000 7fef8d7fa700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fef780387d0 0x7fef7803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.731+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fef8004c9b0 con 0x7fef90074d60 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.732+0000 7fef8f7fe700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fef780387d0 0x7fef7803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:52.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.732+0000 7fef8f7fe700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fef780387d0 0x7fef7803ac90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef84006fd0 tx=0x7fef84006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:52.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:52.737+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef80022770 con 0x7fef90074d60 2024-03-26T21:55:53.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.005+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef7c005190 con 0x7fef90074d60 2024-03-26T21:55:53.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.006+0000 7fef8d7fa700 1 -- 172.21.15.196:0/1552590038 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fef80021840 con 0x7fef90074d60 2024-03-26T21:55:53.002 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:53.003 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fef780387d0 msgr2=0x7fef7803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fef780387d0 0x7fef7803ac90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef84006fd0 tx=0x7fef84006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 msgr2=0x7fef90197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef90197d80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fef80006010 tx=0x7fef800050b0 comp rx=0 tx=0).stop 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 shutdown_connections 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fef780387d0 0x7fef7803ac90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:53.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 --2- 172.21.15.196:0/1552590038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef90074d60 0x7fef90197d80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:53.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 >> 172.21.15.196:0/1552590038 conn(0x7fef900fbd00 msgr2=0x7fef900fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:53.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 shutdown_connections 2024-03-26T21:55:53.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:53.008+0000 7fef96a36700 1 -- 172.21.15.196:0/1552590038 wait complete. 2024-03-26T21:55:53.006 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:54.018 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:53 smithi194 ceph-mon[29693]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:54.018 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:53 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1552590038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:54.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:54.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:54.757 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:54.803 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:55.411 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:55 smithi194 ceph-mon[29693]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:56.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/1920786172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 msgr2=0x7f7d380ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 --2- 172.21.15.196:0/1920786172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d380ffa40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009910 tx=0x7f7d28009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/1920786172 shutdown_connections 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 --2- 172.21.15.196:0/1920786172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d380ffa40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/1920786172 >> 172.21.15.196:0/1920786172 conn(0x7f7d380fabc0 msgr2=0x7f7d380fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/1920786172 shutdown_connections 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/1920786172 wait complete. 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 Processor -- start 2024-03-26T21:55:56.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.280+0000 7f7d3e5a3700 1 -- start start 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.281+0000 7f7d3e5a3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d38194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.281+0000 7f7d3e5a3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38195010 con 0x7f7d380ff620 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.281+0000 7f7d37fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d38194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.281+0000 7f7d37fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d38194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:33722/0 (socket says 172.21.15.196:33722) 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.281+0000 7f7d37fff700 1 -- 172.21.15.196:0/2303363600 learned_addr learned my addr 172.21.15.196:0/2303363600 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.281+0000 7f7d37fff700 1 -- 172.21.15.196:0/2303363600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280095f0 con 0x7f7d380ff620 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.282+0000 7f7d37fff700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d38194ad0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7d28006010 tx=0x7f7d280050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.282+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d2801d070 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.282+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d2800be20 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.282+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d38195210 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.282+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d28022de0 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.282+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d381956b0 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.283+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3818e450 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.283+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f7d280226c0 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.283+0000 7f7d357fa700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7d200387b0 0x7f7d2003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.283+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d2804ca70 con 0x7f7d380ff620 2024-03-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.283+0000 7f7d377fe700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7d200387b0 0x7f7d2003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:56.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.284+0000 7f7d377fe700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7d200387b0 0x7f7d2003ac70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006fd0 tx=0x7f7d2c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:56.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.288+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d2802ba00 con 0x7f7d380ff620 2024-03-26T21:55:56.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.556+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d380623e0 con 0x7f7d380ff620 2024-03-26T21:55:56.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.557+0000 7f7d357fa700 1 -- 172.21.15.196:0/2303363600 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7d380623e0 con 0x7f7d380ff620 2024-03-26T21:55:56.553 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:55:56.554 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:55:56.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7d200387b0 msgr2=0x7f7d2003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7d200387b0 0x7f7d2003ac70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006fd0 tx=0x7f7d2c006e40 comp rx=0 tx=0).stop 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 msgr2=0x7f7d38194ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d38194ad0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7d28006010 tx=0x7f7d280050b0 comp rx=0 tx=0).stop 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 shutdown_connections 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7d200387b0 0x7f7d2003ac70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 --2- 172.21.15.196:0/2303363600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d380ff620 0x7f7d38194ad0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 >> 172.21.15.196:0/2303363600 conn(0x7f7d380fabc0 msgr2=0x7f7d380fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 shutdown_connections 2024-03-26T21:55:56.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:56.559+0000 7f7d3e5a3700 1 -- 172.21.15.196:0/2303363600 wait complete. 2024-03-26T21:55:56.557 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:55:56.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:56 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2303363600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:55:57.911 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:57 smithi194 ceph-mon[29693]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:55:58.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:55:58.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:55:58.339 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:58.385 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:55:59.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 -- 172.21.15.196:0/4279311596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 msgr2=0x7f113c100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 --2- 172.21.15.196:0/4279311596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c100b30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f112c009910 tx=0x7f112c009c20 comp rx=0 tx=0).stop 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 -- 172.21.15.196:0/4279311596 shutdown_connections 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 --2- 172.21.15.196:0/4279311596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c100b30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 -- 172.21.15.196:0/4279311596 >> 172.21.15.196:0/4279311596 conn(0x7f113c0fbcd0 msgr2=0x7f113c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 -- 172.21.15.196:0/4279311596 shutdown_connections 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.732+0000 7f1142531700 1 -- 172.21.15.196:0/4279311596 wait complete. 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f1142531700 1 Processor -- start 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f1142531700 1 -- start start 2024-03-26T21:55:59.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f1142531700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:59.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f1142531700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c074f60 con 0x7f113c100710 2024-03-26T21:55:59.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f113bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:59.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f113bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:33732/0 (socket says 172.21.15.196:33732) 2024-03-26T21:55:59.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.733+0000 7f113bfff700 1 -- 172.21.15.196:0/638683360 learned_addr learned my addr 172.21.15.196:0/638683360 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:55:59.732 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f113bfff700 1 -- 172.21.15.196:0/638683360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0095f0 con 0x7f113c100710 2024-03-26T21:55:59.732 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f113bfff700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c074a20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f112c006010 tx=0x7f112c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112c01d070 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c0731a0 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f112c005dc0 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112c022e90 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.734+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c073640 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.735+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c106e70 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.735+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f112c00b960 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.735+0000 7f11397fa700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f11240387c0 0x7f112403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.735+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f112c04c870 con 0x7f113c100710 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.735+0000 7f113b7fe700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f11240387c0 0x7f112403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:55:59.733 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.736+0000 7f113b7fe700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f11240387c0 0x7f112403ac80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1130006fd0 tx=0x7f1130006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:55:59.736 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:55:59.740+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f112c022750 con 0x7f113c100710 2024-03-26T21:55:59.958 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:55:59 smithi194 ceph-mon[29693]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:00.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.005+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f113c0623e0 con 0x7f113c100710 2024-03-26T21:56:00.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.006+0000 7f11397fa700 1 -- 172.21.15.196:0/638683360 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f112c027020 con 0x7f113c100710 2024-03-26T21:56:00.002 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:00.003 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f11240387c0 msgr2=0x7f112403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f11240387c0 0x7f112403ac80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1130006fd0 tx=0x7f1130006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 msgr2=0x7f113c074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c074a20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f112c006010 tx=0x7f112c009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 shutdown_connections 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f11240387c0 0x7f112403ac80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 --2- 172.21.15.196:0/638683360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f113c100710 0x7f113c074a20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 >> 172.21.15.196:0/638683360 conn(0x7f113c0fbcd0 msgr2=0x7f113c0fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 shutdown_connections 2024-03-26T21:56:00.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:00.008+0000 7f1142531700 1 -- 172.21.15.196:0/638683360 wait complete. 2024-03-26T21:56:00.006 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:01.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:00 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/638683360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:01.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:01.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:01.733 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:01.778 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:02.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:01 smithi194 ceph-mon[29693]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 -- 172.21.15.196:0/3029843143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 msgr2=0x7f3764100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 --2- 172.21.15.196:0/3029843143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764100b30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3754009910 tx=0x7f3754009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 -- 172.21.15.196:0/3029843143 shutdown_connections 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 --2- 172.21.15.196:0/3029843143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764100b30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 -- 172.21.15.196:0/3029843143 >> 172.21.15.196:0/3029843143 conn(0x7f37640fbcd0 msgr2=0x7f37640fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 -- 172.21.15.196:0/3029843143 shutdown_connections 2024-03-26T21:56:03.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.304+0000 7f376924d700 1 -- 172.21.15.196:0/3029843143 wait complete. 2024-03-26T21:56:03.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.305+0000 7f376924d700 1 Processor -- start 2024-03-26T21:56:03.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.305+0000 7f376924d700 1 -- start start 2024-03-26T21:56:03.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.305+0000 7f376924d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.305+0000 7f376924d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37641960f0 con 0x7f3764100710 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.305+0000 7f3762d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f3762d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764195bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57004/0 (socket says 172.21.15.196:57004) 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f3762d9d700 1 -- 172.21.15.196:0/1052293333 learned_addr learned my addr 172.21.15.196:0/1052293333 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f3762d9d700 1 -- 172.21.15.196:0/1052293333 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540095f0 con 0x7f3764100710 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f3762d9d700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764195bb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3754006010 tx=0x7f3754009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f375401d070 con 0x7f3764100710 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3754005dc0 con 0x7f3764100710 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37641962f0 con 0x7f3764100710 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3754022e90 con 0x7f3764100710 2024-03-26T21:56:03.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.306+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3764196790 con 0x7f3764100710 2024-03-26T21:56:03.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.307+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f375400f4f0 con 0x7f3764100710 2024-03-26T21:56:03.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.307+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376418f540 con 0x7f3764100710 2024-03-26T21:56:03.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.307+0000 7f374bfff700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f374c038760 0x7f374c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:03.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.307+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f375404cc00 con 0x7f3764100710 2024-03-26T21:56:03.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.308+0000 7f376259c700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f374c038760 0x7f374c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:03.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.308+0000 7f376259c700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f374c038760 0x7f374c03ac20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3758006fd0 tx=0x7f3758006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:03.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.313+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f375402a540 con 0x7f3764100710 2024-03-26T21:56:03.582 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.586+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37640623e0 con 0x7f3764100710 2024-03-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.586+0000 7f374bfff700 1 -- 172.21.15.196:0/1052293333 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f37640623e0 con 0x7f3764100710 2024-03-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:03.585 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.588+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f374c038760 msgr2=0x7f374c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:03.585 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.588+0000 7f376924d700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f374c038760 0x7f374c03ac20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3758006fd0 tx=0x7f3758006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:03.585 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.588+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 msgr2=0x7f3764195bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:03.585 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.588+0000 7f376924d700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764195bb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3754006010 tx=0x7f3754009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:03.585 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.589+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 shutdown_connections 2024-03-26T21:56:03.585 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.589+0000 7f376924d700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f374c038760 0x7f374c03ac20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:03.586 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.589+0000 7f376924d700 1 --2- 172.21.15.196:0/1052293333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3764100710 0x7f3764195bb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:03.586 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.589+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 >> 172.21.15.196:0/1052293333 conn(0x7f37640fbcd0 msgr2=0x7f37640fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:03.586 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.589+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 shutdown_connections 2024-03-26T21:56:03.586 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:03.589+0000 7f376924d700 1 -- 172.21.15.196:0/1052293333 wait complete. 2024-03-26T21:56:03.586 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:04.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:03 smithi194 ceph-mon[29693]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:04.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:03 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1052293333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:05.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:05.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:05.365 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:05.411 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:05.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:05 smithi194 ceph-mon[29693]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:06.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 -- 172.21.15.196:0/4024909616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 msgr2=0x7f6c38103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 --2- 172.21.15.196:0/4024909616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38103660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009910 tx=0x7f6c2c009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 -- 172.21.15.196:0/4024909616 shutdown_connections 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 --2- 172.21.15.196:0/4024909616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38103660 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 -- 172.21.15.196:0/4024909616 >> 172.21.15.196:0/4024909616 conn(0x7f6c380fab80 msgr2=0x7f6c380fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 -- 172.21.15.196:0/4024909616 shutdown_connections 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.881+0000 7f6c3e47c700 1 -- 172.21.15.196:0/4024909616 wait complete. 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.882+0000 7f6c3e47c700 1 Processor -- start 2024-03-26T21:56:06.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.882+0000 7f6c3e47c700 1 -- start start 2024-03-26T21:56:06.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.882+0000 7f6c3e47c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:06.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.882+0000 7f6c3e47c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c38100fe0 con 0x7f6c38101270 2024-03-26T21:56:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c37fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c37fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57016/0 (socket says 172.21.15.196:57016) 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c37fff700 1 -- 172.21.15.196:0/389008454 learned_addr learned my addr 172.21.15.196:0/389008454 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c37fff700 1 -- 172.21.15.196:0/389008454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c0095f0 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c37fff700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38100aa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009ed0 tx=0x7f6c2c004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c2c01d070 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c380ff220 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c2c00be20 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c2c00f460 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.883+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c380ff6c0 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.884+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6c2c00b960 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.884+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c38106e90 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.884+0000 7f6c357fa700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6c200387d0 0x7f6c2003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.884+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6c2c04c9f0 con 0x7f6c38101270 2024-03-26T21:56:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.885+0000 7f6c377fe700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6c200387d0 0x7f6c2003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:06.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.885+0000 7f6c377fe700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6c200387d0 0x7f6c2003ac90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6c28006fd0 tx=0x7f6c28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:06.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:06.890+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6c2c02a360 con 0x7f6c38101270 2024-03-26T21:56:07.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.157+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c380623e0 con 0x7f6c38101270 2024-03-26T21:56:07.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.158+0000 7f6c357fa700 1 -- 172.21.15.196:0/389008454 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6c380623e0 con 0x7f6c38101270 2024-03-26T21:56:07.155 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:07.155 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6c200387d0 msgr2=0x7f6c2003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6c200387d0 0x7f6c2003ac90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6c28006fd0 tx=0x7f6c28006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 msgr2=0x7f6c38100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38100aa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009ed0 tx=0x7f6c2c004690 comp rx=0 tx=0).stop 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 shutdown_connections 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6c200387d0 0x7f6c2003ac90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 --2- 172.21.15.196:0/389008454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c38101270 0x7f6c38100aa0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 >> 172.21.15.196:0/389008454 conn(0x7f6c380fab80 msgr2=0x7f6c380fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 shutdown_connections 2024-03-26T21:56:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:07.160+0000 7f6c3e47c700 1 -- 172.21.15.196:0/389008454 wait complete. 2024-03-26T21:56:07.158 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:07 smithi194 ceph-mon[29693]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:07 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/389008454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:08.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:08.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:08.919 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:08.965 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:09.964 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:09 smithi194 ceph-mon[29693]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:10.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 -- 172.21.15.196:0/3297065266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 msgr2=0x7f54a80fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:10.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 --2- 172.21.15.196:0/3297065266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a80fee50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5498009910 tx=0x7f5498009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:10.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 -- 172.21.15.196:0/3297065266 shutdown_connections 2024-03-26T21:56:10.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 --2- 172.21.15.196:0/3297065266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a80fee50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:10.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 -- 172.21.15.196:0/3297065266 >> 172.21.15.196:0/3297065266 conn(0x7f54a80fa370 msgr2=0x7f54a80fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:10.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 -- 172.21.15.196:0/3297065266 shutdown_connections 2024-03-26T21:56:10.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.547+0000 7f54acd60700 1 -- 172.21.15.196:0/3297065266 wait complete. 2024-03-26T21:56:10.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.548+0000 7f54acd60700 1 Processor -- start 2024-03-26T21:56:10.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.548+0000 7f54acd60700 1 -- start start 2024-03-26T21:56:10.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.548+0000 7f54acd60700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a8195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:10.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.548+0000 7f54acd60700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81960b0 con 0x7f54a80fea30 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.549+0000 7f54a659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a8195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.549+0000 7f54a659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a8195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57030/0 (socket says 172.21.15.196:57030) 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.549+0000 7f54a659c700 1 -- 172.21.15.196:0/2748570979 learned_addr learned my addr 172.21.15.196:0/2748570979 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.549+0000 7f54a659c700 1 -- 172.21.15.196:0/2748570979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980095f0 con 0x7f54a80fea30 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.549+0000 7f54a659c700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a8195b70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5498006010 tx=0x7f5498009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.549+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f549801d070 con 0x7f54a80fea30 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.550+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8196310 con 0x7f54a80fea30 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.550+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5498005dc0 con 0x7f54a80fea30 2024-03-26T21:56:10.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.550+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5498022e90 con 0x7f54a80fea30 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.550+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a8198fe0 con 0x7f54a80fea30 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.550+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a818f520 con 0x7f54a80fea30 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.551+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f549800b960 con 0x7f54a80fea30 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.551+0000 7f549ffff700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5494038760 0x7f549403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.551+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f549804c7d0 con 0x7f54a80fea30 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.551+0000 7f549dd9b700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5494038760 0x7f549403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:10.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.552+0000 7f549dd9b700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5494038760 0x7f549403ac20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5490006fd0 tx=0x7f5490006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:10.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.556+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f549802a540 con 0x7f54a80fea30 2024-03-26T21:56:10.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.827+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54a80623e0 con 0x7f54a80fea30 2024-03-26T21:56:10.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.827+0000 7f549ffff700 1 -- 172.21.15.196:0/2748570979 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f54a80623e0 con 0x7f54a80fea30 2024-03-26T21:56:10.824 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:10.824 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:10.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5494038760 msgr2=0x7f549403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:10.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5494038760 0x7f549403ac20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5490006fd0 tx=0x7f5490006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:10.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 msgr2=0x7f54a8195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:10.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a8195b70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5498006010 tx=0x7f5498009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:10.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 shutdown_connections 2024-03-26T21:56:10.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5494038760 0x7f549403ac20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:10.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 --2- 172.21.15.196:0/2748570979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54a80fea30 0x7f54a8195b70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:10.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.829+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 >> 172.21.15.196:0/2748570979 conn(0x7f54a80fa370 msgr2=0x7f54a80fae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:10.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.830+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 shutdown_connections 2024-03-26T21:56:10.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:10.830+0000 7f54acd60700 1 -- 172.21.15.196:0/2748570979 wait complete. 2024-03-26T21:56:10.827 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:12.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:11 smithi194 ceph-mon[29693]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:12.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:11 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2748570979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:12.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:12.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:12.602 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:12.647 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:14.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 -- 172.21.15.196:0/3157879089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 msgr2=0x7fc3fc0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 --2- 172.21.15.196:0/3157879089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0ffa40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009910 tx=0x7fc3f8009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 -- 172.21.15.196:0/3157879089 shutdown_connections 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 --2- 172.21.15.196:0/3157879089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0ffa40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 -- 172.21.15.196:0/3157879089 >> 172.21.15.196:0/3157879089 conn(0x7fc3fc0fabc0 msgr2=0x7fc3fc0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 -- 172.21.15.196:0/3157879089 shutdown_connections 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 -- 172.21.15.196:0/3157879089 wait complete. 2024-03-26T21:56:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.105+0000 7fc403a04700 1 Processor -- start 2024-03-26T21:56:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc403a04700 1 -- start start 2024-03-26T21:56:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc403a04700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc403a04700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc069a20 con 0x7fc3fc0ff620 2024-03-26T21:56:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc4017a0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc4017a0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:40004/0 (socket says 172.21.15.196:40004) 2024-03-26T21:56:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc4017a0700 1 -- 172.21.15.196:0/492247679 learned_addr learned my addr 172.21.15.196:0/492247679 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:14.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.106+0000 7fc4017a0700 1 -- 172.21.15.196:0/492247679 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80095f0 con 0x7fc3fc0ff620 2024-03-26T21:56:14.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.107+0000 7fc4017a0700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0694e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8006010 tx=0x7fc3f8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.107+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3f801d070 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.107+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc198920 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.107+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3f8005dc0 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.107+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3f8022e90 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.107+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc198d60 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.108+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fc3f8022470 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.108+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc18e450 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.108+0000 7fc3f27fc700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc3e8038760 0x7fc3e803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.108+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc3f804c8b0 con 0x7fc3fc0ff620 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.108+0000 7fc400f9f700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc3e8038760 0x7fc3e803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:14.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.109+0000 7fc400f9f700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc3e8038760 0x7fc3e803ac20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec006fd0 tx=0x7fc3ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:14.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.113+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc3f800b960 con 0x7fc3fc0ff620 2024-03-26T21:56:14.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:13 smithi194 ceph-mon[29693]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:14.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.384+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3fc0623e0 con 0x7fc3fc0ff620 2024-03-26T21:56:14.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.385+0000 7fc3f27fc700 1 -- 172.21.15.196:0/492247679 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc3f8027030 con 0x7fc3fc0ff620 2024-03-26T21:56:14.381 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:14.382 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc3e8038760 msgr2=0x7fc3e803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc3e8038760 0x7fc3e803ac20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec006fd0 tx=0x7fc3ec006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 msgr2=0x7fc3fc0694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0694e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8006010 tx=0x7fc3f8009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 shutdown_connections 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc3e8038760 0x7fc3e803ac20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 --2- 172.21.15.196:0/492247679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3fc0ff620 0x7fc3fc0694e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 >> 172.21.15.196:0/492247679 conn(0x7fc3fc0fabc0 msgr2=0x7fc3fc0fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 shutdown_connections 2024-03-26T21:56:14.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:14.387+0000 7fc403a04700 1 -- 172.21.15.196:0/492247679 wait complete. 2024-03-26T21:56:14.385 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:15.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:15 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/492247679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:15.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:15 smithi194 ceph-mon[29693]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:15.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:15.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:16.146 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:16.192 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 -- 172.21.15.196:0/437650057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 msgr2=0x7f58a40ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 --2- 172.21.15.196:0/437650057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40ffa70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5898009910 tx=0x7f5898009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 -- 172.21.15.196:0/437650057 shutdown_connections 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 --2- 172.21.15.196:0/437650057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40ffa70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 -- 172.21.15.196:0/437650057 >> 172.21.15.196:0/437650057 conn(0x7f58a40fabf0 msgr2=0x7f58a40fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 -- 172.21.15.196:0/437650057 shutdown_connections 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.693+0000 7f58aa528700 1 -- 172.21.15.196:0/437650057 wait complete. 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.694+0000 7f58aa528700 1 Processor -- start 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.694+0000 7f58aa528700 1 -- start start 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.694+0000 7f58aa528700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:17.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.694+0000 7f58aa528700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4069a20 con 0x7f58a40ff650 2024-03-26T21:56:17.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:17.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:40018/0 (socket says 172.21.15.196:40018) 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a3fff700 1 -- 172.21.15.196:0/1133252383 learned_addr learned my addr 172.21.15.196:0/1133252383 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a3fff700 1 -- 172.21.15.196:0/1133252383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58980095f0 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a3fff700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40694e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5898006010 tx=0x7f5898009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f589801d070 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a4196700 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5898005dc0 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5898022e90 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.695+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a419af60 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.696+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f589800b960 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.696+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a418e480 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.696+0000 7f58a17fa700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f588c038760 0x7f588c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.696+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f589804c8a0 con 0x7f58a40ff650 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.697+0000 7f58a37fe700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f588c038760 0x7f588c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:17.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.697+0000 7f58a37fe700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f588c038760 0x7f588c03ac20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5894006fd0 tx=0x7f5894006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:17.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.702+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f589802a540 con 0x7f58a40ff650 2024-03-26T21:56:17.969 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.973+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58a40623e0 con 0x7f58a40ff650 2024-03-26T21:56:17.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.974+0000 7f58a17fa700 1 -- 172.21.15.196:0/1133252383 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5898027020 con 0x7f58a40ff650 2024-03-26T21:56:17.970 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:17.971 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:17.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f588c038760 msgr2=0x7f588c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f588c038760 0x7f588c03ac20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5894006fd0 tx=0x7f5894006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 msgr2=0x7f58a40694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40694e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5898006010 tx=0x7f5898009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 shutdown_connections 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f588c038760 0x7f588c03ac20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 --2- 172.21.15.196:0/1133252383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a40ff650 0x7f58a40694e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 >> 172.21.15.196:0/1133252383 conn(0x7f58a40fabf0 msgr2=0x7f58a40fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 shutdown_connections 2024-03-26T21:56:17.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:17.976+0000 7f58aa528700 1 -- 172.21.15.196:0/1133252383 wait complete. 2024-03-26T21:56:17.974 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:18.161 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:17 smithi194 ceph-mon[29693]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:19.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:18 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1133252383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:19.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:19.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:19.731 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:19.778 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:19.984 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:19 smithi194 ceph-mon[29693]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 -- 172.21.15.196:0/3820949641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 msgr2=0x7fcecc0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 --2- 172.21.15.196:0/3820949641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0ffa50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009910 tx=0x7fcec0009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 -- 172.21.15.196:0/3820949641 shutdown_connections 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 --2- 172.21.15.196:0/3820949641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0ffa50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 -- 172.21.15.196:0/3820949641 >> 172.21.15.196:0/3820949641 conn(0x7fcecc0fabd0 msgr2=0x7fcecc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 -- 172.21.15.196:0/3820949641 shutdown_connections 2024-03-26T21:56:21.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.152+0000 7fced15d4700 1 -- 172.21.15.196:0/3820949641 wait complete. 2024-03-26T21:56:21.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.153+0000 7fced15d4700 1 Processor -- start 2024-03-26T21:56:21.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.153+0000 7fced15d4700 1 -- start start 2024-03-26T21:56:21.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.153+0000 7fced15d4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0fca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:21.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.153+0000 7fced15d4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0fcf80 con 0x7fcecc0ff630 2024-03-26T21:56:21.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fcecaffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:21.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fcecaffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0fca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:40040/0 (socket says 172.21.15.196:40040) 2024-03-26T21:56:21.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fcecaffd700 1 -- 172.21.15.196:0/1050369448 learned_addr learned my addr 172.21.15.196:0/1050369448 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:21.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fcecaffd700 1 -- 172.21.15.196:0/1050369448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec00095f0 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fcecaffd700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0fca40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcec0006010 tx=0x7fcec0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec001d070 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcec0005dc0 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc0fb1c0 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec0022e90 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.154+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc0fb660 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.155+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc105d90 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.155+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fcec000b960 con 0x7fcecc0ff630 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.155+0000 7fceb3fff700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fceb4038760 0x7fceb403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:21.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.155+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcec004c880 con 0x7fcecc0ff630 2024-03-26T21:56:21.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.156+0000 7fceca7fc700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fceb4038760 0x7fceb403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:21.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.156+0000 7fceca7fc700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fceb4038760 0x7fceb403ac20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcebc006fd0 tx=0x7fcebc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:21.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.161+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcec0022750 con 0x7fcecc0ff630 2024-03-26T21:56:21.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.429+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcecc0623e0 con 0x7fcecc0ff630 2024-03-26T21:56:21.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.430+0000 7fceb3fff700 1 -- 172.21.15.196:0/1050369448 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fcec0027020 con 0x7fcecc0ff630 2024-03-26T21:56:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:21.427 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:21.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fceb4038760 msgr2=0x7fceb403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fceb4038760 0x7fceb403ac20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcebc006fd0 tx=0x7fcebc006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 msgr2=0x7fcecc0fca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0fca40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcec0006010 tx=0x7fcec0009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 shutdown_connections 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fceb4038760 0x7fceb403ac20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 --2- 172.21.15.196:0/1050369448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcecc0ff630 0x7fcecc0fca40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 >> 172.21.15.196:0/1050369448 conn(0x7fcecc0fabd0 msgr2=0x7fcecc068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:21.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 shutdown_connections 2024-03-26T21:56:21.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:21.432+0000 7fced15d4700 1 -- 172.21.15.196:0/1050369448 wait complete. 2024-03-26T21:56:21.430 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:22.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:21 smithi194 ceph-mon[29693]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:22.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:21 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1050369448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:23.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:23.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:23.228 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:23.275 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:24.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:23 smithi194 ceph-mon[29693]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:24.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 -- 172.21.15.196:0/2889461130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 msgr2=0x7f3c68100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:24.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 --2- 172.21.15.196:0/2889461130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68100b40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009910 tx=0x7f3c58009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 -- 172.21.15.196:0/2889461130 shutdown_connections 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 --2- 172.21.15.196:0/2889461130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68100b40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 -- 172.21.15.196:0/2889461130 >> 172.21.15.196:0/2889461130 conn(0x7f3c680fbcc0 msgr2=0x7f3c680fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 -- 172.21.15.196:0/2889461130 shutdown_connections 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.606+0000 7f3c6d705700 1 -- 172.21.15.196:0/2889461130 wait complete. 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c6d705700 1 Processor -- start 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c6d705700 1 -- start start 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c6d705700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c6d705700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681982f0 con 0x7f3c68100720 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c66ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:24.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c66ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56240/0 (socket says 172.21.15.196:56240) 2024-03-26T21:56:24.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c66ffd700 1 -- 172.21.15.196:0/1652692593 learned_addr learned my addr 172.21.15.196:0/1652692593 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:24.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.607+0000 7f3c66ffd700 1 -- 172.21.15.196:0/1652692593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c580095f0 con 0x7f3c68100720 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.608+0000 7f3c66ffd700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68197db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3c58006010 tx=0x7f3c58009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.608+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c5801d070 con 0x7f3c68100720 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.608+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c681984f0 con 0x7f3c68100720 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.608+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c58005dc0 con 0x7f3c68100720 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.608+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c58022e90 con 0x7f3c68100720 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.608+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c68198990 con 0x7f3c68100720 2024-03-26T21:56:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.609+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c68191950 con 0x7f3c68100720 2024-03-26T21:56:24.608 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.609+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3c58022470 con 0x7f3c68100720 2024-03-26T21:56:24.608 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.609+0000 7f3c4ffff700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3c50038760 0x7f3c5003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:24.608 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.609+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3c5804c8b0 con 0x7f3c68100720 2024-03-26T21:56:24.608 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.609+0000 7f3c667fc700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3c50038760 0x7f3c5003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:24.608 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.610+0000 7f3c667fc700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3c50038760 0x7f3c5003ac20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c006fd0 tx=0x7f3c5c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:24.611 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.614+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3c5800b960 con 0x7f3c68100720 2024-03-26T21:56:24.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.883+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c680623e0 con 0x7f3c68100720 2024-03-26T21:56:24.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.884+0000 7f3c4ffff700 1 -- 172.21.15.196:0/1652692593 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3c680623e0 con 0x7f3c68100720 2024-03-26T21:56:24.881 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:24.881 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3c50038760 msgr2=0x7f3c5003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3c50038760 0x7f3c5003ac20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c006fd0 tx=0x7f3c5c006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 msgr2=0x7f3c68197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68197db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3c58006010 tx=0x7f3c58009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 shutdown_connections 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3c50038760 0x7f3c5003ac20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 --2- 172.21.15.196:0/1652692593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c68100720 0x7f3c68197db0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 >> 172.21.15.196:0/1652692593 conn(0x7f3c680fbcc0 msgr2=0x7f3c680fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 shutdown_connections 2024-03-26T21:56:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:24.886+0000 7f3c6d705700 1 -- 172.21.15.196:0/1652692593 wait complete. 2024-03-26T21:56:24.884 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:25.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:25 smithi194 ceph-mon[29693]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:25.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:25 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1652692593' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:26.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:26.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:26.609 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:26.655 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:28.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.064+0000 7fc03093f700 1 -- 172.21.15.196:0/286601248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 msgr2=0x7fc02c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:28.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.064+0000 7fc03093f700 1 --2- 172.21.15.196:0/286601248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c100b40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc020009910 tx=0x7fc020009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:28.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 -- 172.21.15.196:0/286601248 shutdown_connections 2024-03-26T21:56:28.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 --2- 172.21.15.196:0/286601248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c100b40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:28.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 -- 172.21.15.196:0/286601248 >> 172.21.15.196:0/286601248 conn(0x7fc02c0fbcc0 msgr2=0x7fc02c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:28.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 -- 172.21.15.196:0/286601248 shutdown_connections 2024-03-26T21:56:28.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 -- 172.21.15.196:0/286601248 wait complete. 2024-03-26T21:56:28.064 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 Processor -- start 2024-03-26T21:56:28.064 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.065+0000 7fc03093f700 1 -- start start 2024-03-26T21:56:28.064 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc03093f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc03093f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c196100 con 0x7fc02c100720 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc02a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc02a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56262/0 (socket says 172.21.15.196:56262) 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc02a59c700 1 -- 172.21.15.196:0/3298729722 learned_addr learned my addr 172.21.15.196:0/3298729722 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc02a59c700 1 -- 172.21.15.196:0/3298729722 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0200095f0 con 0x7fc02c100720 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.066+0000 7fc02a59c700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c195bc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc020006010 tx=0x7fc020009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc02001d070 con 0x7fc02c100720 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c196300 con 0x7fc02c100720 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc020005dc0 con 0x7fc02c100720 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc020022e90 con 0x7fc02c100720 2024-03-26T21:56:28.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c1967a0 con 0x7fc02c100720 2024-03-26T21:56:28.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fc02000b960 con 0x7fc02c100720 2024-03-26T21:56:28.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.067+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c18f550 con 0x7fc02c100720 2024-03-26T21:56:28.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.068+0000 7fc01b7fe700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc014038760 0x7fc01403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:28.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.068+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc02004c800 con 0x7fc02c100720 2024-03-26T21:56:28.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.068+0000 7fc029d9b700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc014038760 0x7fc01403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:28.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.068+0000 7fc029d9b700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc014038760 0x7fc01403ac20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc01c006fd0 tx=0x7fc01c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:28.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.073+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc020022750 con 0x7fc02c100720 2024-03-26T21:56:28.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:27 smithi194 ceph-mon[29693]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:28.341 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.345+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc02c0623e0 con 0x7fc02c100720 2024-03-26T21:56:28.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.345+0000 7fc01b7fe700 1 -- 172.21.15.196:0/3298729722 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc02c0623e0 con 0x7fc02c100720 2024-03-26T21:56:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:28.342 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc014038760 msgr2=0x7fc01403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc014038760 0x7fc01403ac20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc01c006fd0 tx=0x7fc01c006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 msgr2=0x7fc02c195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c195bc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc020006010 tx=0x7fc020009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 shutdown_connections 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc014038760 0x7fc01403ac20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 --2- 172.21.15.196:0/3298729722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100720 0x7fc02c195bc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:28.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 >> 172.21.15.196:0/3298729722 conn(0x7fc02c0fbcc0 msgr2=0x7fc02c0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:28.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.347+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 shutdown_connections 2024-03-26T21:56:28.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:28.348+0000 7fc03093f700 1 -- 172.21.15.196:0/3298729722 wait complete. 2024-03-26T21:56:28.345 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:29.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:28 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3298729722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:29.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:29.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:29.997 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:29 smithi194 ceph-mon[29693]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:30.166 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:30.212 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:31.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 -- 172.21.15.196:0/2347945740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 msgr2=0x7efc0c0ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:31.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 --2- 172.21.15.196:0/2347945740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c0ff3d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009910 tx=0x7efbfc009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:31.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 -- 172.21.15.196:0/2347945740 shutdown_connections 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 --2- 172.21.15.196:0/2347945740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c0ff3d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 -- 172.21.15.196:0/2347945740 >> 172.21.15.196:0/2347945740 conn(0x7efc0c0fabd0 msgr2=0x7efc0c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 -- 172.21.15.196:0/2347945740 shutdown_connections 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 -- 172.21.15.196:0/2347945740 wait complete. 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 Processor -- start 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.714+0000 7efc124a5700 1 -- start start 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc124a5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc124a5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c198320 con 0x7efc0c0fefb0 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc0bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:31.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc0bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56280/0 (socket says 172.21.15.196:56280) 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc0bfff700 1 -- 172.21.15.196:0/3989906672 learned_addr learned my addr 172.21.15.196:0/3989906672 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc0bfff700 1 -- 172.21.15.196:0/3989906672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0095f0 con 0x7efc0c0fefb0 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.715+0000 7efc0bfff700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c197de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009ed0 tx=0x7efbfc004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.716+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbfc01d070 con 0x7efc0c0fefb0 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.716+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0c198520 con 0x7efc0c0fefb0 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.716+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbfc00be20 con 0x7efc0c0fefb0 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.716+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbfc00f460 con 0x7efc0c0fefb0 2024-03-26T21:56:31.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.716+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0c1989c0 con 0x7efc0c0fefb0 2024-03-26T21:56:31.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.717+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0c191750 con 0x7efc0c0fefb0 2024-03-26T21:56:31.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.717+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7efbfc022800 con 0x7efc0c0fefb0 2024-03-26T21:56:31.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.717+0000 7efc097fa700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbf40387d0 0x7efbf403ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:31.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.717+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efbfc04cc00 con 0x7efc0c0fefb0 2024-03-26T21:56:31.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.717+0000 7efc0b7fe700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbf40387d0 0x7efbf403ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:31.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.717+0000 7efc0b7fe700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbf40387d0 0x7efbf403ac90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc00006fd0 tx=0x7efc00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:31.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.722+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efbfc053080 con 0x7efc0c0fefb0 2024-03-26T21:56:31.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.997+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc0c0623e0 con 0x7efc0c0fefb0 2024-03-26T21:56:31.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:31.998+0000 7efc097fa700 1 -- 172.21.15.196:0/3989906672 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7efbfc02bba0 con 0x7efc0c0fefb0 2024-03-26T21:56:31.994 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:31.994 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:31.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbf40387d0 msgr2=0x7efbf403ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbf40387d0 0x7efbf403ac90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc00006fd0 tx=0x7efc00006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 msgr2=0x7efc0c197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c197de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009ed0 tx=0x7efbfc004690 comp rx=0 tx=0).stop 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 shutdown_connections 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbf40387d0 0x7efbf403ac90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 --2- 172.21.15.196:0/3989906672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc0c0fefb0 0x7efc0c197de0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 >> 172.21.15.196:0/3989906672 conn(0x7efc0c0fabd0 msgr2=0x7efc0c0fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 shutdown_connections 2024-03-26T21:56:31.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:32.000+0000 7efc124a5700 1 -- 172.21.15.196:0/3989906672 wait complete. 2024-03-26T21:56:31.998 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:32.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:31 smithi194 ceph-mon[29693]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:33.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:32 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3989906672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:33.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:33.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:33.803 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:33.849 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:34.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:33 smithi194 ceph-mon[29693]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:35.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:35 smithi194 ceph-mon[29693]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 -- 172.21.15.196:0/1557588244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 msgr2=0x7f37fc100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 --2- 172.21.15.196:0/1557588244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc100980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009910 tx=0x7f37f0009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 -- 172.21.15.196:0/1557588244 shutdown_connections 2024-03-26T21:56:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 --2- 172.21.15.196:0/1557588244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc100980 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:35.425 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 -- 172.21.15.196:0/1557588244 >> 172.21.15.196:0/1557588244 conn(0x7f37fc0fbb20 msgr2=0x7f37fc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:35.425 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 -- 172.21.15.196:0/1557588244 shutdown_connections 2024-03-26T21:56:35.425 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.427+0000 7f380122d700 1 -- 172.21.15.196:0/1557588244 wait complete. 2024-03-26T21:56:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.428+0000 7f380122d700 1 Processor -- start 2024-03-26T21:56:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.428+0000 7f380122d700 1 -- start start 2024-03-26T21:56:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.428+0000 7f380122d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.428+0000 7f380122d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc198140 con 0x7f37fc100560 2024-03-26T21:56:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37fbfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37fbfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:54868/0 (socket says 172.21.15.196:54868) 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37fbfff700 1 -- 172.21.15.196:0/856336623 learned_addr learned my addr 172.21.15.196:0/856336623 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37fbfff700 1 -- 172.21.15.196:0/856336623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f00095f0 con 0x7f37fc100560 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37fbfff700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc197c00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37f0006010 tx=0x7f37f0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f001d070 con 0x7f37fc100560 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc198340 con 0x7f37fc100560 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37f0005dc0 con 0x7f37fc100560 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f0022e90 con 0x7f37fc100560 2024-03-26T21:56:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.429+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc1987e0 con 0x7f37fc100560 2024-03-26T21:56:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.430+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc191790 con 0x7f37fc100560 2024-03-26T21:56:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.430+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f37f0022470 con 0x7f37fc100560 2024-03-26T21:56:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.431+0000 7f37f97fa700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f37e4038760 0x7f37e403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.431+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f37f004c9e0 con 0x7f37fc100560 2024-03-26T21:56:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.431+0000 7f37fb7fe700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f37e4038760 0x7f37e403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.431+0000 7f37fb7fe700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f37e4038760 0x7f37e403ac20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006fd0 tx=0x7f37ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:35.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.436+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f37f000b960 con 0x7f37fc100560 2024-03-26T21:56:35.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.706+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37fc0623e0 con 0x7f37fc100560 2024-03-26T21:56:35.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.706+0000 7f37f97fa700 1 -- 172.21.15.196:0/856336623 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f37fc0623e0 con 0x7f37fc100560 2024-03-26T21:56:35.703 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:35.703 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:35.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f37e4038760 msgr2=0x7f37e403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f37e4038760 0x7f37e403ac20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006fd0 tx=0x7f37ec006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 msgr2=0x7f37fc197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc197c00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37f0006010 tx=0x7f37f0009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 shutdown_connections 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f37e4038760 0x7f37e403ac20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 --2- 172.21.15.196:0/856336623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37fc100560 0x7f37fc197c00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 >> 172.21.15.196:0/856336623 conn(0x7f37fc0fbb20 msgr2=0x7f37fc0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 shutdown_connections 2024-03-26T21:56:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:35.709+0000 7f380122d700 1 -- 172.21.15.196:0/856336623 wait complete. 2024-03-26T21:56:35.707 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:36.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:36 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/856336623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:37.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:37.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:37.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:37 smithi194 ceph-mon[29693]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:37.495 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:37.540 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.054+0000 7effac6c2700 1 -- 172.21.15.196:0/2839522253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 msgr2=0x7effa40ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.054+0000 7effac6c2700 1 --2- 172.21.15.196:0/2839522253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa40ffa30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00b0a0 tx=0x7eff9c00b3b0 comp rx=0 tx=0).stop 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.054+0000 7effac6c2700 1 -- 172.21.15.196:0/2839522253 shutdown_connections 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.054+0000 7effac6c2700 1 --2- 172.21.15.196:0/2839522253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa40ffa30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.054+0000 7effac6c2700 1 -- 172.21.15.196:0/2839522253 >> 172.21.15.196:0/2839522253 conn(0x7effa40fabb0 msgr2=0x7effa40fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effac6c2700 1 -- 172.21.15.196:0/2839522253 shutdown_connections 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effac6c2700 1 -- 172.21.15.196:0/2839522253 wait complete. 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effac6c2700 1 Processor -- start 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effac6c2700 1 -- start start 2024-03-26T21:56:39.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effac6c2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa4069180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effac6c2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40696c0 con 0x7effa40ff610 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effaa45e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa4069180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effaa45e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa4069180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:54886/0 (socket says 172.21.15.196:54886) 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.055+0000 7effaa45e700 1 -- 172.21.15.196:0/3320810970 learned_addr learned my addr 172.21.15.196:0/3320810970 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7effaa45e700 1 -- 172.21.15.196:0/3320810970 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c009d00 con 0x7effa40ff610 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7effaa45e700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa4069180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0062a0 tx=0x7eff9c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff9c00e040 con 0x7effa40ff610 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa41988e0 con 0x7effa40ff610 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff9c003d70 con 0x7effa40ff610 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff9c01c890 con 0x7effa40ff610 2024-03-26T21:56:39.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.056+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa4198d80 con 0x7effa40ff610 2024-03-26T21:56:39.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.057+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7eff9c01cb50 con 0x7effa40ff610 2024-03-26T21:56:39.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.057+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa418e440 con 0x7effa40ff610 2024-03-26T21:56:39.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.057+0000 7eff9b7fe700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7eff900387c0 0x7eff9003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:39.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.057+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7eff9c04ca60 con 0x7effa40ff610 2024-03-26T21:56:39.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.057+0000 7effa9c5d700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7eff900387c0 0x7eff9003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:39.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.058+0000 7effa9c5d700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7eff900387c0 0x7eff9003ac80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff94006fd0 tx=0x7eff94006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:39.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.062+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff9c01a3b0 con 0x7effa40ff610 2024-03-26T21:56:39.333 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.336+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effa40623e0 con 0x7effa40ff610 2024-03-26T21:56:39.333 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.337+0000 7eff9b7fe700 1 -- 172.21.15.196:0/3320810970 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7effa40623e0 con 0x7effa40ff610 2024-03-26T21:56:39.333 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:39.333 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:39.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7eff900387c0 msgr2=0x7eff9003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:39.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7eff900387c0 0x7eff9003ac80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff94006fd0 tx=0x7eff94006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 msgr2=0x7effa4069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa4069180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0062a0 tx=0x7eff9c00bd40 comp rx=0 tx=0).stop 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 shutdown_connections 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7eff900387c0 0x7eff9003ac80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 --2- 172.21.15.196:0/3320810970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effa40ff610 0x7effa4069180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 >> 172.21.15.196:0/3320810970 conn(0x7effa40fabb0 msgr2=0x7effa40fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 shutdown_connections 2024-03-26T21:56:39.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:39.339+0000 7effac6c2700 1 -- 172.21.15.196:0/3320810970 wait complete. 2024-03-26T21:56:39.337 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:40.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:39 smithi194 ceph-mon[29693]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:40.010 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:39 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3320810970' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:40.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:40.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:41.070 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:41.115 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:42.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:41 smithi194 ceph-mon[29693]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:42.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 -- 172.21.15.196:0/482001944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 msgr2=0x7f581c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 --2- 172.21.15.196:0/482001944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c0ffa40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b0a0 tx=0x7f580c00b3b0 comp rx=0 tx=0).stop 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 -- 172.21.15.196:0/482001944 shutdown_connections 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 --2- 172.21.15.196:0/482001944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c0ffa40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 -- 172.21.15.196:0/482001944 >> 172.21.15.196:0/482001944 conn(0x7f581c0fabc0 msgr2=0x7f581c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 -- 172.21.15.196:0/482001944 shutdown_connections 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.635+0000 7f5821e49700 1 -- 172.21.15.196:0/482001944 wait complete. 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f5821e49700 1 Processor -- start 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f5821e49700 1 -- start start 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f5821e49700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:42.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f5821e49700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c196110 con 0x7f581c0ff620 2024-03-26T21:56:42.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f581b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:42.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f581b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36032/0 (socket says 172.21.15.196:36032) 2024-03-26T21:56:42.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f581b7fe700 1 -- 172.21.15.196:0/3567228962 learned_addr learned my addr 172.21.15.196:0/3567228962 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.636+0000 7f581b7fe700 1 -- 172.21.15.196:0/3567228962 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c009d00 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.637+0000 7f581b7fe700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c195bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f580c0062a0 tx=0x7f580c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.637+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580c00e040 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.637+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f580c003d70 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.637+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c196310 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.637+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580c01c890 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.637+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c1967b0 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.638+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f580c01cb50 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.638+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c18f550 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.638+0000 7f5818ff9700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f58040387c0 0x7f580403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.638+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f580c04c9a0 con 0x7f581c0ff620 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.638+0000 7f581affd700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f58040387c0 0x7f580403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:42.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.639+0000 7f581affd700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f58040387c0 0x7f580403ac80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5810006fd0 tx=0x7f5810006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:42.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.643+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f580c01a3b0 con 0x7f581c0ff620 2024-03-26T21:56:42.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.915+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f581c0623e0 con 0x7f581c0ff620 2024-03-26T21:56:42.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.916+0000 7f5818ff9700 1 -- 172.21.15.196:0/3567228962 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f581c0623e0 con 0x7f581c0ff620 2024-03-26T21:56:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:42.913 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:42.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f58040387c0 msgr2=0x7f580403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f58040387c0 0x7f580403ac80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5810006fd0 tx=0x7f5810006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 msgr2=0x7f581c195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c195bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f580c0062a0 tx=0x7f580c00bd40 comp rx=0 tx=0).stop 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 shutdown_connections 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f58040387c0 0x7f580403ac80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 --2- 172.21.15.196:0/3567228962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581c0ff620 0x7f581c195bd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 >> 172.21.15.196:0/3567228962 conn(0x7f581c0fabc0 msgr2=0x7f581c0fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 shutdown_connections 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:42.918+0000 7f5821e49700 1 -- 172.21.15.196:0/3567228962 wait complete. 2024-03-26T21:56:42.915 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:44.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:43 smithi194 ceph-mon[29693]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:44.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:43 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3567228962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:44.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:44.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:44.691 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:44.737 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:45.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:45 smithi194 ceph-mon[29693]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- 172.21.15.196:0/768751194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 msgr2=0x7f7bf00ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 --2- 172.21.15.196:0/768751194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00ffa40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009910 tx=0x7f7bec009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- 172.21.15.196:0/768751194 shutdown_connections 2024-03-26T21:56:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 --2- 172.21.15.196:0/768751194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00ffa40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- 172.21.15.196:0/768751194 >> 172.21.15.196:0/768751194 conn(0x7f7bf00fab80 msgr2=0x7f7bf00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- 172.21.15.196:0/768751194 shutdown_connections 2024-03-26T21:56:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- 172.21.15.196:0/768751194 wait complete. 2024-03-26T21:56:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 Processor -- start 2024-03-26T21:56:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- start start 2024-03-26T21:56:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.196+0000 7f7bf8002700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00fcf70 con 0x7f7bf00ff620 2024-03-26T21:56:46.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf5d9e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:46.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf5d9e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36040/0 (socket says 172.21.15.196:36040) 2024-03-26T21:56:46.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf5d9e700 1 -- 172.21.15.196:0/1878148776 learned_addr learned my addr 172.21.15.196:0/1878148776 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:46.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf5d9e700 1 -- 172.21.15.196:0/1878148776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec0095f0 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf5d9e700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00fca30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7bec006010 tx=0x7f7bec0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bec01d070 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf00fb1b0 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bec00be20 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bec022de0 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.197+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf00fb650 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.198+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0105d80 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.198+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f7bec0226c0 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.198+0000 7f7be6ffd700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7bdc0387b0 0x7f7bdc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.198+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7bec04cc30 con 0x7f7bf00ff620 2024-03-26T21:56:46.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.199+0000 7f7bf559d700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7bdc0387b0 0x7f7bdc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:46.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.199+0000 7f7bf559d700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7bdc0387b0 0x7f7bdc03ac70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7be0006fd0 tx=0x7f7be0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:46.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.204+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7bec02ba00 con 0x7f7bf00ff620 2024-03-26T21:56:46.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.475+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bf00623e0 con 0x7f7bf00ff620 2024-03-26T21:56:46.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.475+0000 7f7be6ffd700 1 -- 172.21.15.196:0/1878148776 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7bec027030 con 0x7f7bf00ff620 2024-03-26T21:56:46.472 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:46.472 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:46.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7bdc0387b0 msgr2=0x7f7bdc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7bdc0387b0 0x7f7bdc03ac70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7be0006fd0 tx=0x7f7be0006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:46.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 msgr2=0x7f7bf00fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00fca30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7bec006010 tx=0x7f7bec0050b0 comp rx=0 tx=0).stop 2024-03-26T21:56:46.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 shutdown_connections 2024-03-26T21:56:46.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7bdc0387b0 0x7f7bdc03ac70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:46.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 --2- 172.21.15.196:0/1878148776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7bf00ff620 0x7f7bf00fca30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:46.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 >> 172.21.15.196:0/1878148776 conn(0x7f7bf00fab80 msgr2=0x7f7bf00682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:46.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 shutdown_connections 2024-03-26T21:56:46.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:46.477+0000 7f7bf8002700 1 -- 172.21.15.196:0/1878148776 wait complete. 2024-03-26T21:56:46.475 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:46.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:46 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1878148776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:47.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:47 smithi194 ceph-mon[29693]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:48.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:48.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:48.211 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:48.256 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.722+0000 7f0fa2809700 1 -- 172.21.15.196:0/1774154755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 msgr2=0x7f0f9c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.722+0000 7f0fa2809700 1 --2- 172.21.15.196:0/1774154755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c100b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009910 tx=0x7f0f8c009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.722+0000 7f0fa2809700 1 -- 172.21.15.196:0/1774154755 shutdown_connections 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.722+0000 7f0fa2809700 1 --2- 172.21.15.196:0/1774154755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c100b40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.722+0000 7f0fa2809700 1 -- 172.21.15.196:0/1774154755 >> 172.21.15.196:0/1774154755 conn(0x7f0f9c0fbcc0 msgr2=0x7f0f9c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.723+0000 7f0fa2809700 1 -- 172.21.15.196:0/1774154755 shutdown_connections 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.723+0000 7f0fa2809700 1 -- 172.21.15.196:0/1774154755 wait complete. 2024-03-26T21:56:49.720 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.723+0000 7f0fa2809700 1 Processor -- start 2024-03-26T21:56:49.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.723+0000 7f0fa2809700 1 -- start start 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.723+0000 7f0fa2809700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.723+0000 7f0fa2809700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c074f60 con 0x7f0f9c100720 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.724+0000 7f0f9bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.724+0000 7f0f9bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36062/0 (socket says 172.21.15.196:36062) 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.724+0000 7f0f9bfff700 1 -- 172.21.15.196:0/2070102724 learned_addr learned my addr 172.21.15.196:0/2070102724 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.724+0000 7f0f9bfff700 1 -- 172.21.15.196:0/2070102724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0095f0 con 0x7f0f9c100720 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.724+0000 7f0f9bfff700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c074a20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c006010 tx=0x7f0f8c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f8c01d070 con 0x7f0f9c100720 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f8c005dc0 con 0x7f0f9c100720 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c0731a0 con 0x7f0f9c100720 2024-03-26T21:56:49.723 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f8c022e90 con 0x7f0f9c100720 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c073640 con 0x7f0f9c100720 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f0f8c00b960 con 0x7f0f9c100720 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.725+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c106e80 con 0x7f0f9c100720 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.726+0000 7f0f997fa700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0f840387c0 0x7f0f8403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.726+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0f8c04c840 con 0x7f0f9c100720 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.726+0000 7f0f9b7fe700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0f840387c0 0x7f0f8403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:49.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.726+0000 7f0f9b7fe700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0f840387c0 0x7f0f8403ac80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f90006fd0 tx=0x7f0f90006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:49.728 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.731+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f8c00bd00 con 0x7f0f9c100720 2024-03-26T21:56:49.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.997+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f9c0623e0 con 0x7f0f9c100720 2024-03-26T21:56:49.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:49.998+0000 7f0f997fa700 1 -- 172.21.15.196:0/2070102724 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f0f8c027020 con 0x7f0f9c100720 2024-03-26T21:56:49.995 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:49.995 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0f840387c0 msgr2=0x7f0f8403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0f840387c0 0x7f0f8403ac80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f90006fd0 tx=0x7f0f90006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 msgr2=0x7f0f9c074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c074a20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c006010 tx=0x7f0f8c009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 shutdown_connections 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0f840387c0 0x7f0f8403ac80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 --2- 172.21.15.196:0/2070102724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f9c100720 0x7f0f9c074a20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 >> 172.21.15.196:0/2070102724 conn(0x7f0f9c0fbcc0 msgr2=0x7f0f9c0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 shutdown_connections 2024-03-26T21:56:49.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:50.001+0000 7f0fa2809700 1 -- 172.21.15.196:0/2070102724 wait complete. 2024-03-26T21:56:49.999 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:50.023 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:49 smithi194 ceph-mon[29693]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:51.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:50 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2070102724' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:51.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:51.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:51.765 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:51.811 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:52.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:51 smithi194 ceph-mon[29693]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:53.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 -- 172.21.15.196:0/496866709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 msgr2=0x7efc04100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 --2- 172.21.15.196:0/496866709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04100b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009910 tx=0x7efbf8009c20 comp rx=0 tx=0).stop 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 -- 172.21.15.196:0/496866709 shutdown_connections 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 --2- 172.21.15.196:0/496866709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04100b40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 -- 172.21.15.196:0/496866709 >> 172.21.15.196:0/496866709 conn(0x7efc040fbcc0 msgr2=0x7efc040fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 -- 172.21.15.196:0/496866709 shutdown_connections 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.327+0000 7efc09692700 1 -- 172.21.15.196:0/496866709 wait complete. 2024-03-26T21:56:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.328+0000 7efc09692700 1 Processor -- start 2024-03-26T21:56:53.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.328+0000 7efc09692700 1 -- start start 2024-03-26T21:56:53.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.328+0000 7efc09692700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:53.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.328+0000 7efc09692700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04074f60 con 0x7efc04100720 2024-03-26T21:56:53.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc02ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:53.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc02ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49840/0 (socket says 172.21.15.196:49840) 2024-03-26T21:56:53.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc02ffd700 1 -- 172.21.15.196:0/4217452180 learned_addr learned my addr 172.21.15.196:0/4217452180 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc02ffd700 1 -- 172.21.15.196:0/4217452180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80095f0 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc02ffd700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04074a20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efbf8006010 tx=0x7efbf8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf801d070 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc040731a0 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbf8005dc0 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf8022e90 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.329+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc04073640 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.330+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7efbf800b960 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.330+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc04106e80 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.330+0000 7efbebfff700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbec038760 0x7efbec03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.331+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efbf804c880 con 0x7efc04100720 2024-03-26T21:56:53.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.331+0000 7efc027fc700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbec038760 0x7efbec03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:53.329 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.331+0000 7efc027fc700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbec038760 0x7efbec03ac20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efbf4006fd0 tx=0x7efbf4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:53.332 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.335+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efbf8022750 con 0x7efc04100720 2024-03-26T21:56:53.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.602+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc040623e0 con 0x7efc04100720 2024-03-26T21:56:53.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.602+0000 7efbebfff700 1 -- 172.21.15.196:0/4217452180 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7efbf8027020 con 0x7efc04100720 2024-03-26T21:56:53.599 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:53.599 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.604+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbec038760 msgr2=0x7efbec03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.604+0000 7efc09692700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbec038760 0x7efbec03ac20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efbf4006fd0 tx=0x7efbf4006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.604+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 msgr2=0x7efc04074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.604+0000 7efc09692700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04074a20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efbf8006010 tx=0x7efbf8009f80 comp rx=0 tx=0).stop 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.605+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 shutdown_connections 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.605+0000 7efc09692700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbec038760 0x7efbec03ac20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.605+0000 7efc09692700 1 --2- 172.21.15.196:0/4217452180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc04100720 0x7efc04074a20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.605+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 >> 172.21.15.196:0/4217452180 conn(0x7efc040fbcc0 msgr2=0x7efc040fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.605+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 shutdown_connections 2024-03-26T21:56:53.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:53.605+0000 7efc09692700 1 -- 172.21.15.196:0/4217452180 wait complete. 2024-03-26T21:56:53.603 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:54.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:53 smithi194 ceph-mon[29693]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:54.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:53 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4217452180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:55.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:55.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:55.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:55 smithi194 ceph-mon[29693]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:55.429 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:56.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 -- 172.21.15.196:0/1051077787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 msgr2=0x7f07480ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:56.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 --2- 172.21.15.196:0/1051077787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480ffa40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f074000b0a0 tx=0x7f074000b3b0 comp rx=0 tx=0).stop 2024-03-26T21:56:56.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 -- 172.21.15.196:0/1051077787 shutdown_connections 2024-03-26T21:56:56.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 --2- 172.21.15.196:0/1051077787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480ffa40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:56.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 -- 172.21.15.196:0/1051077787 >> 172.21.15.196:0/1051077787 conn(0x7f07480fab80 msgr2=0x7f07480fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:56.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 -- 172.21.15.196:0/1051077787 shutdown_connections 2024-03-26T21:56:56.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.839+0000 7f074fbf5700 1 -- 172.21.15.196:0/1051077787 wait complete. 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074fbf5700 1 Processor -- start 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074fbf5700 1 -- start start 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074fbf5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074fbf5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07480fcf70 con 0x7f07480ff620 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074d991700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074d991700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49858/0 (socket says 172.21.15.196:49858) 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074d991700 1 -- 172.21.15.196:0/2834040614 learned_addr learned my addr 172.21.15.196:0/2834040614 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.840+0000 7f074d991700 1 -- 172.21.15.196:0/2834040614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0740009d00 con 0x7f07480ff620 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.841+0000 7f074d991700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480fca30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07400062a0 tx=0x7f074000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.841+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f074000e040 con 0x7f07480ff620 2024-03-26T21:56:56.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.841+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07480fb1b0 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.841+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0740003d70 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.841+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f074001c890 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.841+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07480fb650 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.842+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f074001cb50 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.842+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0748105d80 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.842+0000 7f073effd700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07340387c0 0x7f073403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.842+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f074004ca00 con 0x7f07480ff620 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.842+0000 7f074d190700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07340387c0 0x7f073403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:56:56.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.843+0000 7f074d190700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07340387c0 0x7f073403ac80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0738006fd0 tx=0x7f0738006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:56:56.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:56.847+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f074001a590 con 0x7f07480ff620 2024-03-26T21:56:57.115 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.118+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f074818f100 con 0x7f07480ff620 2024-03-26T21:56:57.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.119+0000 7f073effd700 1 -- 172.21.15.196:0/2834040614 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f074001a3b0 con 0x7f07480ff620 2024-03-26T21:56:57.116 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:56:57.116 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:56:57.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07340387c0 msgr2=0x7f073403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07340387c0 0x7f073403ac80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0738006fd0 tx=0x7f0738006e40 comp rx=0 tx=0).stop 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 msgr2=0x7f07480fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480fca30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07400062a0 tx=0x7f074000bd40 comp rx=0 tx=0).stop 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 shutdown_connections 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07340387c0 0x7f073403ac80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 --2- 172.21.15.196:0/2834040614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07480ff620 0x7f07480fca30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 >> 172.21.15.196:0/2834040614 conn(0x7f07480fab80 msgr2=0x7f07480682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 shutdown_connections 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:56:57.121+0000 7f074fbf5700 1 -- 172.21.15.196:0/2834040614 wait complete. 2024-03-26T21:56:57.119 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:56:58.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:57 smithi194 ceph-mon[29693]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:56:58.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:57 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2834040614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:56:58.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:56:58.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:56:58.904 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:56:58.949 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:00.036 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:56:59 smithi194 ceph-mon[29693]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:00.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 -- 172.21.15.196:0/3252356675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 msgr2=0x7f6428100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:00.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 --2- 172.21.15.196:0/3252356675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428100b60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6418009910 tx=0x7f6418009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:00.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 -- 172.21.15.196:0/3252356675 shutdown_connections 2024-03-26T21:57:00.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 --2- 172.21.15.196:0/3252356675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428100b60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:00.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 -- 172.21.15.196:0/3252356675 >> 172.21.15.196:0/3252356675 conn(0x7f64280fbd00 msgr2=0x7f64280fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:00.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 -- 172.21.15.196:0/3252356675 shutdown_connections 2024-03-26T21:57:00.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.500+0000 7f642d7c7700 1 -- 172.21.15.196:0/3252356675 wait complete. 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f642d7c7700 1 Processor -- start 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f642d7c7700 1 -- start start 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f642d7c7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f642d7c7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6428196100 con 0x7f6428100740 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f6426ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f6426ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49884/0 (socket says 172.21.15.196:49884) 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.501+0000 7f6426ffd700 1 -- 172.21.15.196:0/2088346539 learned_addr learned my addr 172.21.15.196:0/2088346539 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f6426ffd700 1 -- 172.21.15.196:0/2088346539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64180095f0 con 0x7f6428100740 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f6426ffd700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428195bc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6418006010 tx=0x7f6418009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:00.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641801d070 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6428196300 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6418005dc0 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6418022e90 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.502+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64281967a0 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.503+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642818f570 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.503+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6418022470 con 0x7f6428100740 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.503+0000 7f640ffff700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6410038760 0x7f641003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:00.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.503+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f641804c7f0 con 0x7f6428100740 2024-03-26T21:57:00.502 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.503+0000 7f64267fc700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6410038760 0x7f641003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:00.502 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.504+0000 7f64267fc700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6410038760 0x7f641003ac20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f641c006fd0 tx=0x7f641c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:00.505 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.508+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f641800f4f0 con 0x7f6428100740 2024-03-26T21:57:00.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.774+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64280623e0 con 0x7f6428100740 2024-03-26T21:57:00.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.775+0000 7f640ffff700 1 -- 172.21.15.196:0/2088346539 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6418027070 con 0x7f6428100740 2024-03-26T21:57:00.772 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:00.772 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:00.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6410038760 msgr2=0x7f641003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:00.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6410038760 0x7f641003ac20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f641c006fd0 tx=0x7f641c006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:00.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 msgr2=0x7f6428195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:00.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428195bc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6418006010 tx=0x7f6418009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:00.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 shutdown_connections 2024-03-26T21:57:00.775 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6410038760 0x7f641003ac20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:00.775 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 --2- 172.21.15.196:0/2088346539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6428100740 0x7f6428195bc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:00.775 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.777+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 >> 172.21.15.196:0/2088346539 conn(0x7f64280fbd00 msgr2=0x7f64280fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:00.775 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.778+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 shutdown_connections 2024-03-26T21:57:00.775 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:00.778+0000 7f642d7c7700 1 -- 172.21.15.196:0/2088346539 wait complete. 2024-03-26T21:57:00.775 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:02.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:01 smithi194 ceph-mon[29693]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:02.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:01 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2088346539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:02.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:02.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:02.615 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:02.662 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:04.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:03 smithi194 ceph-mon[29693]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:04.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 -- 172.21.15.196:0/1363794186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 msgr2=0x7f3924100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:04.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 --2- 172.21.15.196:0/1363794186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924100b40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3914009910 tx=0x7f3914009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:04.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 -- 172.21.15.196:0/1363794186 shutdown_connections 2024-03-26T21:57:04.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 --2- 172.21.15.196:0/1363794186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924100b40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:04.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 -- 172.21.15.196:0/1363794186 >> 172.21.15.196:0/1363794186 conn(0x7f39240fbcc0 msgr2=0x7f39240fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 -- 172.21.15.196:0/1363794186 shutdown_connections 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 -- 172.21.15.196:0/1363794186 wait complete. 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 Processor -- start 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.194+0000 7f39295e4700 1 -- start start 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f39295e4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f39295e4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3924196100 con 0x7f3924100720 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f3922ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f3922ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36508/0 (socket says 172.21.15.196:36508) 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f3922ffd700 1 -- 172.21.15.196:0/3263735286 learned_addr learned my addr 172.21.15.196:0/3263735286 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f3922ffd700 1 -- 172.21.15.196:0/3263735286 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39140095f0 con 0x7f3924100720 2024-03-26T21:57:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.195+0000 7f3922ffd700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924195bc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3914006010 tx=0x7f3914009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:04.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.196+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f391401d070 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.196+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3924196300 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.196+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3914005dc0 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.196+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3914022e90 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.196+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39241967a0 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.197+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392418f550 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.197+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3914022470 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.197+0000 7f390bfff700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f390c038760 0x7f390c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.197+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f391404c7f0 con 0x7f3924100720 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.197+0000 7f39227fc700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f390c038760 0x7f390c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:04.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.197+0000 7f39227fc700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f390c038760 0x7f390c03ac20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3918006fd0 tx=0x7f3918006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:04.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.202+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f391400f4f0 con 0x7f3924100720 2024-03-26T21:57:04.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.472+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39240623e0 con 0x7f3924100720 2024-03-26T21:57:04.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.472+0000 7f390bfff700 1 -- 172.21.15.196:0/3263735286 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3914027070 con 0x7f3924100720 2024-03-26T21:57:04.469 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:04.469 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:04.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f390c038760 msgr2=0x7f390c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:04.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f390c038760 0x7f390c03ac20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3918006fd0 tx=0x7f3918006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:04.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 msgr2=0x7f3924195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924195bc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3914006010 tx=0x7f3914009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 shutdown_connections 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f390c038760 0x7f390c03ac20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 --2- 172.21.15.196:0/3263735286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3924100720 0x7f3924195bc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.474+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 >> 172.21.15.196:0/3263735286 conn(0x7f39240fbcc0 msgr2=0x7f39240fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.475+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 shutdown_connections 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:04.475+0000 7f39295e4700 1 -- 172.21.15.196:0/3263735286 wait complete. 2024-03-26T21:57:04.472 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:05.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:05 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3263735286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:05.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:05 smithi194 ceph-mon[29693]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:06.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:06.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:06.234 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:06.278 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 -- 172.21.15.196:0/2775982243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 msgr2=0x7f0a8c103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 --2- 172.21.15.196:0/2775982243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c103660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009910 tx=0x7f0a80009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 -- 172.21.15.196:0/2775982243 shutdown_connections 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 --2- 172.21.15.196:0/2775982243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c103660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 -- 172.21.15.196:0/2775982243 >> 172.21.15.196:0/2775982243 conn(0x7f0a8c0fab80 msgr2=0x7f0a8c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 -- 172.21.15.196:0/2775982243 shutdown_connections 2024-03-26T21:57:07.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.788+0000 7f0a925c6700 1 -- 172.21.15.196:0/2775982243 wait complete. 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.789+0000 7f0a925c6700 1 Processor -- start 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.789+0000 7f0a925c6700 1 -- start start 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.789+0000 7f0a925c6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.789+0000 7f0a925c6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c1960b0 con 0x7f0a8c101270 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a8bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a8bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36522/0 (socket says 172.21.15.196:36522) 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a8bfff700 1 -- 172.21.15.196:0/3427543019 learned_addr learned my addr 172.21.15.196:0/3427543019 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a8bfff700 1 -- 172.21.15.196:0/3427543019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a800095f0 con 0x7f0a8c101270 2024-03-26T21:57:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a8bfff700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c195b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0a80006010 tx=0x7f0a800050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a8001d070 con 0x7f0a8c101270 2024-03-26T21:57:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a8000be20 con 0x7f0a8c101270 2024-03-26T21:57:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c1962b0 con 0x7f0a8c101270 2024-03-26T21:57:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a8000f460 con 0x7f0a8c101270 2024-03-26T21:57:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.790+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c196750 con 0x7f0a8c101270 2024-03-26T21:57:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.791+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f0a8000b960 con 0x7f0a8c101270 2024-03-26T21:57:07.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.791+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c18f560 con 0x7f0a8c101270 2024-03-26T21:57:07.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.791+0000 7f0a897fa700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0a74038820 0x7f0a7403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:07.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.791+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0a8004c990 con 0x7f0a8c101270 2024-03-26T21:57:07.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.792+0000 7f0a8b7fe700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0a74038820 0x7f0a7403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:07.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.792+0000 7f0a8b7fe700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0a74038820 0x7f0a7403ace0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c006fd0 tx=0x7f0a7c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:07.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:07.797+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0a80005c00 con 0x7f0a8c101270 2024-03-26T21:57:08.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.069+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a8c0623e0 con 0x7f0a8c101270 2024-03-26T21:57:08.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.070+0000 7f0a897fa700 1 -- 172.21.15.196:0/3427543019 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f0a8c0623e0 con 0x7f0a8c101270 2024-03-26T21:57:08.067 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:08.067 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0a74038820 msgr2=0x7f0a7403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0a74038820 0x7f0a7403ace0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c006fd0 tx=0x7f0a7c006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 msgr2=0x7f0a8c195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c195b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0a80006010 tx=0x7f0a800050b0 comp rx=0 tx=0).stop 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 shutdown_connections 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0a74038820 0x7f0a7403ace0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:08.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 --2- 172.21.15.196:0/3427543019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8c101270 0x7f0a8c195b70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:08.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 >> 172.21.15.196:0/3427543019 conn(0x7f0a8c0fab80 msgr2=0x7f0a8c0fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:08.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 shutdown_connections 2024-03-26T21:57:08.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:08.072+0000 7f0a925c6700 1 -- 172.21.15.196:0/3427543019 wait complete. 2024-03-26T21:57:08.070 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:08.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:07 smithi194 ceph-mon[29693]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:09.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:08 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3427543019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:09.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:09.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:09.836 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:09.882 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:10.049 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:09 smithi194 ceph-mon[29693]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:11.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 -- 172.21.15.196:0/2794232576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 msgr2=0x7ff158100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:11.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 --2- 172.21.15.196:0/2794232576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158100b40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff148009910 tx=0x7ff148009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 -- 172.21.15.196:0/2794232576 shutdown_connections 2024-03-26T21:57:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 --2- 172.21.15.196:0/2794232576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158100b40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 -- 172.21.15.196:0/2794232576 >> 172.21.15.196:0/2794232576 conn(0x7ff1580fbcc0 msgr2=0x7ff1580fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 -- 172.21.15.196:0/2794232576 shutdown_connections 2024-03-26T21:57:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.348+0000 7ff15e03f700 1 -- 172.21.15.196:0/2794232576 wait complete. 2024-03-26T21:57:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff15e03f700 1 Processor -- start 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff15e03f700 1 -- start start 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff15e03f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff15e03f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff158196100 con 0x7ff158100720 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff1577fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff1577fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36544/0 (socket says 172.21.15.196:36544) 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.349+0000 7ff1577fe700 1 -- 172.21.15.196:0/3408272191 learned_addr learned my addr 172.21.15.196:0/3408272191 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:11.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff1577fe700 1 -- 172.21.15.196:0/3408272191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1480095f0 con 0x7ff158100720 2024-03-26T21:57:11.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff1577fe700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158195bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff148006010 tx=0x7ff148009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:11.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff14801d070 con 0x7ff158100720 2024-03-26T21:57:11.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff158196300 con 0x7ff158100720 2024-03-26T21:57:11.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1581967a0 con 0x7ff158100720 2024-03-26T21:57:11.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff148005dc0 con 0x7ff158100720 2024-03-26T21:57:11.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.350+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff148022e70 con 0x7ff158100720 2024-03-26T21:57:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.351+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff14800f4b0 con 0x7ff158100720 2024-03-26T21:57:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.351+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15818f550 con 0x7ff158100720 2024-03-26T21:57:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.351+0000 7ff154ff9700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff140038770 0x7ff14003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.351+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff14804c880 con 0x7ff158100720 2024-03-26T21:57:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.351+0000 7ff156ffd700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff140038770 0x7ff14003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.352+0000 7ff156ffd700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff140038770 0x7ff14003ac30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff14c006fd0 tx=0x7ff14c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:11.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.356+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff14802a360 con 0x7ff158100720 2024-03-26T21:57:11.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.624+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1580623e0 con 0x7ff158100720 2024-03-26T21:57:11.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.625+0000 7ff154ff9700 1 -- 172.21.15.196:0/3408272191 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff1580623e0 con 0x7ff158100720 2024-03-26T21:57:11.622 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:11.622 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:11.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.626+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff140038770 msgr2=0x7ff14003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff140038770 0x7ff14003ac30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff14c006fd0 tx=0x7ff14c006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 msgr2=0x7ff158195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158195bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff148006010 tx=0x7ff148009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 shutdown_connections 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff140038770 0x7ff14003ac30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 --2- 172.21.15.196:0/3408272191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff158100720 0x7ff158195bc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 >> 172.21.15.196:0/3408272191 conn(0x7ff1580fbcc0 msgr2=0x7ff1580fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 shutdown_connections 2024-03-26T21:57:11.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:11.627+0000 7ff15e03f700 1 -- 172.21.15.196:0/3408272191 wait complete. 2024-03-26T21:57:11.625 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:12.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:11 smithi194 ceph-mon[29693]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:12.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:11 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3408272191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:13.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:13.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:13.469 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:13.515 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:14.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:13 smithi194 ceph-mon[29693]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:14.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.909+0000 7f796a963700 1 -- 172.21.15.196:0/2850684377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 msgr2=0x7f79640ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.909+0000 7f796a963700 1 --2- 172.21.15.196:0/2850684377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f79640ff9d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7954009910 tx=0x7f7954009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 -- 172.21.15.196:0/2850684377 shutdown_connections 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 --2- 172.21.15.196:0/2850684377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f79640ff9d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 -- 172.21.15.196:0/2850684377 >> 172.21.15.196:0/2850684377 conn(0x7f79640fab50 msgr2=0x7f79640fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 -- 172.21.15.196:0/2850684377 shutdown_connections 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 -- 172.21.15.196:0/2850684377 wait complete. 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 Processor -- start 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 -- start start 2024-03-26T21:57:14.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.910+0000 7f796a963700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f7964195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.911+0000 7f796a963700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7964196140 con 0x7f79640ff5b0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.911+0000 7f7963fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f7964195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.911+0000 7f7963fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f7964195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:52818/0 (socket says 172.21.15.196:52818) 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.911+0000 7f7963fff700 1 -- 172.21.15.196:0/1916879561 learned_addr learned my addr 172.21.15.196:0/1916879561 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.911+0000 7f7963fff700 1 -- 172.21.15.196:0/1916879561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79540095f0 con 0x7f79640ff5b0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.911+0000 7f7963fff700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f7964195c00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7954004f40 tx=0x7f7954004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.912+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f795401d070 con 0x7f79640ff5b0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.912+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7964196340 con 0x7f79640ff5b0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.912+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7954005dc0 con 0x7f79640ff5b0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.912+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7954022e40 con 0x7f79640ff5b0 2024-03-26T21:57:14.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.912+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79641967e0 con 0x7f79640ff5b0 2024-03-26T21:57:14.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.912+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796418f700 con 0x7f79640ff5b0 2024-03-26T21:57:14.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.913+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f79540226c0 con 0x7f79640ff5b0 2024-03-26T21:57:14.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.913+0000 7f79617fa700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f794c038760 0x7f794c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:14.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.913+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f795404c8c0 con 0x7f79640ff5b0 2024-03-26T21:57:14.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.913+0000 7f79637fe700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f794c038760 0x7f794c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:14.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.913+0000 7f79637fe700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f794c038760 0x7f794c03ac20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7958006fd0 tx=0x7f7958006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:14.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:14.918+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f795400b960 con 0x7f79640ff5b0 2024-03-26T21:57:15.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.185+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79640623e0 con 0x7f79640ff5b0 2024-03-26T21:57:15.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.186+0000 7f79617fa700 1 -- 172.21.15.196:0/1916879561 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f79640623e0 con 0x7f79640ff5b0 2024-03-26T21:57:15.183 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:15.184 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.188+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f794c038760 msgr2=0x7f794c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.188+0000 7f796a963700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f794c038760 0x7f794c03ac20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7958006fd0 tx=0x7f7958006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 msgr2=0x7f7964195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f7964195c00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7954004f40 tx=0x7f7954004740 comp rx=0 tx=0).stop 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 shutdown_connections 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f794c038760 0x7f794c03ac20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 --2- 172.21.15.196:0/1916879561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79640ff5b0 0x7f7964195c00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 >> 172.21.15.196:0/1916879561 conn(0x7f79640fab50 msgr2=0x7f7964068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 shutdown_connections 2024-03-26T21:57:15.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:15.189+0000 7f796a963700 1 -- 172.21.15.196:0/1916879561 wait complete. 2024-03-26T21:57:15.187 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:15.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:15 smithi194 ceph-mon[29693]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:16.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:16 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1916879561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:16.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:16.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:16.987 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:17.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:17 smithi194 ceph-mon[29693]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:18.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 -- 172.21.15.196:0/3441266369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 msgr2=0x7f97ec100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 --2- 172.21.15.196:0/3441266369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec100b40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009910 tx=0x7f97dc009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 -- 172.21.15.196:0/3441266369 shutdown_connections 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 --2- 172.21.15.196:0/3441266369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec100b40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 -- 172.21.15.196:0/3441266369 >> 172.21.15.196:0/3441266369 conn(0x7f97ec0fbcc0 msgr2=0x7f97ec0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 -- 172.21.15.196:0/3441266369 shutdown_connections 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 -- 172.21.15.196:0/3441266369 wait complete. 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.489+0000 7f97f194b700 1 Processor -- start 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97f194b700 1 -- start start 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97f194b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:18.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97f194b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec196100 con 0x7f97ec100720 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:52844/0 (socket says 172.21.15.196:52844) 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97eaffd700 1 -- 172.21.15.196:0/3140339485 learned_addr learned my addr 172.21.15.196:0/3140339485 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97eaffd700 1 -- 172.21.15.196:0/3140339485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc0095f0 con 0x7f97ec100720 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97eaffd700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec195bc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97dc006010 tx=0x7f97dc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.490+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97dc01d070 con 0x7f97ec100720 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.491+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97dc005dc0 con 0x7f97ec100720 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.491+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97dc022e90 con 0x7f97ec100720 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.491+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec196300 con 0x7f97ec100720 2024-03-26T21:57:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.491+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec1967a0 con 0x7f97ec100720 2024-03-26T21:57:18.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.491+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec18f550 con 0x7f97ec100720 2024-03-26T21:57:18.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.491+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f97dc022470 con 0x7f97ec100720 2024-03-26T21:57:18.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.492+0000 7f97f0949700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d4038760 0x7f97d403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:18.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.492+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f97dc04c920 con 0x7f97ec100720 2024-03-26T21:57:18.494 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.492+0000 7f97ea7fc700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d4038760 0x7f97d403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:18.494 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.492+0000 7f97ea7fc700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d4038760 0x7f97d403ac20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006fd0 tx=0x7f97e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:18.494 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.497+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97dc00b960 con 0x7f97ec100720 2024-03-26T21:57:18.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.764+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97ec0623e0 con 0x7f97ec100720 2024-03-26T21:57:18.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.765+0000 7f97f0949700 1 -- 172.21.15.196:0/3140339485 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f97ec0623e0 con 0x7f97ec100720 2024-03-26T21:57:18.762 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:18.762 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:18.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d4038760 msgr2=0x7f97d403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:18.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d4038760 0x7f97d403ac20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006fd0 tx=0x7f97e0006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:18.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 msgr2=0x7f97ec195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec195bc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97dc006010 tx=0x7f97dc009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 shutdown_connections 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d4038760 0x7f97d403ac20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 --2- 172.21.15.196:0/3140339485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97ec100720 0x7f97ec195bc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 >> 172.21.15.196:0/3140339485 conn(0x7f97ec0fbcc0 msgr2=0x7f97ec0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 shutdown_connections 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:18.767+0000 7f97f194b700 1 -- 172.21.15.196:0/3140339485 wait complete. 2024-03-26T21:57:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:20.063 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:19 smithi194 ceph-mon[29693]: pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:20.063 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:19 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3140339485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:20.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:20.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:20.557 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:20.603 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 -- 172.21.15.196:0/1537875032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 msgr2=0x7f85880731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 --2- 172.21.15.196:0/1537875032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f85880731e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8578009910 tx=0x7f8578009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 -- 172.21.15.196:0/1537875032 shutdown_connections 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 --2- 172.21.15.196:0/1537875032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f85880731e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 -- 172.21.15.196:0/1537875032 >> 172.21.15.196:0/1537875032 conn(0x7f85880fbcc0 msgr2=0x7f85880fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 -- 172.21.15.196:0/1537875032 shutdown_connections 2024-03-26T21:57:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.012+0000 7f858ddf2700 1 -- 172.21.15.196:0/1537875032 wait complete. 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.013+0000 7f858ddf2700 1 Processor -- start 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.013+0000 7f858ddf2700 1 -- start start 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.013+0000 7f858ddf2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f8588197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.013+0000 7f858ddf2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85881982e0 con 0x7f8588074d80 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f85877fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f8588197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f85877fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f8588197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:52876/0 (socket says 172.21.15.196:52876) 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f85877fe700 1 -- 172.21.15.196:0/1681997386 learned_addr learned my addr 172.21.15.196:0/1681997386 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f85877fe700 1 -- 172.21.15.196:0/1681997386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780095f0 con 0x7f8588074d80 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f85877fe700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f8588197da0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8578009ed0 tx=0x7f8578004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:22.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f857801d070 con 0x7f8588074d80 2024-03-26T21:57:22.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85881984e0 con 0x7f8588074d80 2024-03-26T21:57:22.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f857800be20 con 0x7f8588074d80 2024-03-26T21:57:22.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.014+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f857800f460 con 0x7f8588074d80 2024-03-26T21:57:22.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.015+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8588198980 con 0x7f8588074d80 2024-03-26T21:57:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.015+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8588191750 con 0x7f8588074d80 2024-03-26T21:57:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.015+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f857800b960 con 0x7f8588074d80 2024-03-26T21:57:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.016+0000 7f8584ff9700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8570038820 0x7f857003ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.016+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f857804c890 con 0x7f8588074d80 2024-03-26T21:57:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.016+0000 7f8586ffd700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8570038820 0x7f857003ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.016+0000 7f8586ffd700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8570038820 0x7f857003ace0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f857c006fd0 tx=0x7f857c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:22.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.021+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f857802a360 con 0x7f8588074d80 2024-03-26T21:57:22.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:21 smithi194 ceph-mon[29693]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:22.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.291+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85880623e0 con 0x7f8588074d80 2024-03-26T21:57:22.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.291+0000 7f8584ff9700 1 -- 172.21.15.196:0/1681997386 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f85880623e0 con 0x7f8588074d80 2024-03-26T21:57:22.288 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:22.289 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8570038820 msgr2=0x7f857003ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8570038820 0x7f857003ace0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f857c006fd0 tx=0x7f857c006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 msgr2=0x7f8588197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f8588197da0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8578009ed0 tx=0x7f8578004690 comp rx=0 tx=0).stop 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 shutdown_connections 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8570038820 0x7f857003ace0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.293+0000 7f858ddf2700 1 --2- 172.21.15.196:0/1681997386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8588074d80 0x7f8588197da0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.294+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 >> 172.21.15.196:0/1681997386 conn(0x7f85880fbcc0 msgr2=0x7f85880fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.294+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 shutdown_connections 2024-03-26T21:57:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:22.294+0000 7f858ddf2700 1 -- 172.21.15.196:0/1681997386 wait complete. 2024-03-26T21:57:22.292 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:23.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:22 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1681997386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:23.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:23.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:24.057 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:24.102 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:24.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:23 smithi194 ceph-mon[29693]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:25.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:25 smithi194 ceph-mon[29693]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.676+0000 7f6dc783f700 1 -- 172.21.15.196:0/1587806736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 msgr2=0x7f6dc0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.676+0000 7f6dc783f700 1 --2- 172.21.15.196:0/1587806736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0100b40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6db800b0a0 tx=0x7f6db800b3b0 comp rx=0 tx=0).stop 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.676+0000 7f6dc783f700 1 -- 172.21.15.196:0/1587806736 shutdown_connections 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.676+0000 7f6dc783f700 1 --2- 172.21.15.196:0/1587806736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0100b40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 -- 172.21.15.196:0/1587806736 >> 172.21.15.196:0/1587806736 conn(0x7f6dc00fbcc0 msgr2=0x7f6dc00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 -- 172.21.15.196:0/1587806736 shutdown_connections 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 -- 172.21.15.196:0/1587806736 wait complete. 2024-03-26T21:57:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 Processor -- start 2024-03-26T21:57:25.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 -- start start 2024-03-26T21:57:25.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:25.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.677+0000 7f6dc783f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0074f60 con 0x7f6dc0100720 2024-03-26T21:57:25.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.678+0000 7f6dc55db700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:25.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.678+0000 7f6dc55db700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42082/0 (socket says 172.21.15.196:42082) 2024-03-26T21:57:25.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.678+0000 7f6dc55db700 1 -- 172.21.15.196:0/1728922199 learned_addr learned my addr 172.21.15.196:0/1728922199 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:25.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.678+0000 7f6dc55db700 1 -- 172.21.15.196:0/1728922199 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8009d00 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.678+0000 7f6dc55db700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0074a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6db80062a0 tx=0x7f6db800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db800e040 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc00731a0 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6db8003d70 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db801c890 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc0073640 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6db801cb50 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.679+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc0106e80 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.680+0000 7f6db67fc700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6dac0387c0 0x7f6dac03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.680+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6db804c960 con 0x7f6dc0100720 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.680+0000 7f6dc4dda700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6dac0387c0 0x7f6dac03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:25.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.680+0000 7f6dc4dda700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6dac0387c0 0x7f6dac03ac80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6db0006fd0 tx=0x7f6db0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:25.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.685+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6db801a590 con 0x7f6dc0100720 2024-03-26T21:57:25.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.954+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6dc00623e0 con 0x7f6dc0100720 2024-03-26T21:57:25.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.954+0000 7f6db67fc700 1 -- 172.21.15.196:0/1728922199 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6db801a3b0 con 0x7f6dc0100720 2024-03-26T21:57:25.952 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:25.952 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:25.953 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6dac0387c0 msgr2=0x7f6dac03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6dac0387c0 0x7f6dac03ac80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6db0006fd0 tx=0x7f6db0006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 msgr2=0x7f6dc0074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0074a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6db80062a0 tx=0x7f6db800bd40 comp rx=0 tx=0).stop 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 shutdown_connections 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6dac0387c0 0x7f6dac03ac80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 --2- 172.21.15.196:0/1728922199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc0100720 0x7f6dc0074a20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 >> 172.21.15.196:0/1728922199 conn(0x7f6dc00fbcc0 msgr2=0x7f6dc00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 shutdown_connections 2024-03-26T21:57:25.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:25.956+0000 7f6dc783f700 1 -- 172.21.15.196:0/1728922199 wait complete. 2024-03-26T21:57:25.955 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:26.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:26 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1728922199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:27.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:27 smithi194 ceph-mon[29693]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:27.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:27.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:27.715 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:27.761 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:29.227 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 -- 172.21.15.196:0/4063771290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 msgr2=0x7f76f40fee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:29.227 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 --2- 172.21.15.196:0/4063771290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f40fee60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009910 tx=0x7f76e4009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 -- 172.21.15.196:0/4063771290 shutdown_connections 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 --2- 172.21.15.196:0/4063771290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f40fee60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 -- 172.21.15.196:0/4063771290 >> 172.21.15.196:0/4063771290 conn(0x7f76f40fa380 msgr2=0x7f76f40fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 -- 172.21.15.196:0/4063771290 shutdown_connections 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.229+0000 7f76f97ee700 1 -- 172.21.15.196:0/4063771290 wait complete. 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f97ee700 1 Processor -- start 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f97ee700 1 -- start start 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f97ee700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f41012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f97ee700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4101830 con 0x7f76f40fea40 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f41012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f41012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42090/0 (socket says 172.21.15.196:42090) 2024-03-26T21:57:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f2ffd700 1 -- 172.21.15.196:0/2254611877 learned_addr learned my addr 172.21.15.196:0/2254611877 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:29.229 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.230+0000 7f76f2ffd700 1 -- 172.21.15.196:0/2254611877 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e40095f0 con 0x7f76f40fea40 2024-03-26T21:57:29.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.231+0000 7f76f2ffd700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f41012f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f76e4006010 tx=0x7f76e4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:29.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.231+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e401d070 con 0x7f76f40fea40 2024-03-26T21:57:29.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.231+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f40ffa70 con 0x7f76f40fea40 2024-03-26T21:57:29.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.231+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76e4005dc0 con 0x7f76f40fea40 2024-03-26T21:57:29.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.231+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e4022e90 con 0x7f76f40fea40 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.231+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f40fff70 con 0x7f76f40fea40 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.232+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f4106e60 con 0x7f76f40fea40 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.232+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f76e4022470 con 0x7f76f40fea40 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.232+0000 7f76f0ff9700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f76e0038760 0x7f76e003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.232+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f76e404c870 con 0x7f76f40fea40 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.232+0000 7f76ea7fc700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f76e0038760 0x7f76e003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:29.231 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.233+0000 7f76ea7fc700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f76e0038760 0x7f76e003ac20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f76dc006fd0 tx=0x7f76dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:29.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.237+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f76e400b960 con 0x7f76f40fea40 2024-03-26T21:57:29.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.504+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76f40623e0 con 0x7f76f40fea40 2024-03-26T21:57:29.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.504+0000 7f76f0ff9700 1 -- 172.21.15.196:0/2254611877 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f76e4027020 con 0x7f76f40fea40 2024-03-26T21:57:29.502 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:29.502 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f76e0038760 msgr2=0x7f76e003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f76e0038760 0x7f76e003ac20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f76dc006fd0 tx=0x7f76dc006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 msgr2=0x7f76f41012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f41012f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f76e4006010 tx=0x7f76e4009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 shutdown_connections 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f76e0038760 0x7f76e003ac20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:29.504 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 --2- 172.21.15.196:0/2254611877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f40fea40 0x7f76f41012f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:29.505 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 >> 172.21.15.196:0/2254611877 conn(0x7f76f40fa380 msgr2=0x7f76f40fae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:29.505 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 shutdown_connections 2024-03-26T21:57:29.505 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:29.507+0000 7f76f97ee700 1 -- 172.21.15.196:0/2254611877 wait complete. 2024-03-26T21:57:29.505 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:30.076 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:29 smithi194 ceph-mon[29693]: pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:30.076 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:29 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2254611877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:31.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:31.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:31.314 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:31.360 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:32.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:31 smithi194 ceph-mon[29693]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:32.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- 172.21.15.196:0/78677202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 msgr2=0x7f81c4100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:32.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 --2- 172.21.15.196:0/78677202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c4100fe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00b0a0 tx=0x7f81bc00b3b0 comp rx=0 tx=0).stop 2024-03-26T21:57:32.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- 172.21.15.196:0/78677202 shutdown_connections 2024-03-26T21:57:32.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 --2- 172.21.15.196:0/78677202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c4100fe0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:32.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- 172.21.15.196:0/78677202 >> 172.21.15.196:0/78677202 conn(0x7f81c40fa370 msgr2=0x7f81c40fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:32.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- 172.21.15.196:0/78677202 shutdown_connections 2024-03-26T21:57:32.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- 172.21.15.196:0/78677202 wait complete. 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 Processor -- start 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- start start 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c40ff220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.750+0000 7f81cac4b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c40ff760 con 0x7f81c4100bc0 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c89e7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c40ff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c89e7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c40ff220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57988/0 (socket says 172.21.15.196:57988) 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c89e7700 1 -- 172.21.15.196:0/3840961555 learned_addr learned my addr 172.21.15.196:0/3840961555 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:32.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c89e7700 1 -- 172.21.15.196:0/3840961555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc009d00 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c89e7700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c40ff220 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0062a0 tx=0x7f81bc00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81bc00e040 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81bc003d70 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c4198950 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.751+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81bc01c890 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.752+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c4198d90 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.752+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c418e460 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.753+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f81bc01cb00 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.753+0000 7f81c1ffb700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81b0038810 0x7f81b003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.753+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f81bc04c8d0 con 0x7f81c4100bc0 2024-03-26T21:57:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.753+0000 7f81c3fff700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81b0038810 0x7f81b003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:32.752 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.753+0000 7f81c3fff700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81b0038810 0x7f81b003acd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81b4006fd0 tx=0x7f81b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:32.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:32.758+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f81bc012640 con 0x7f81c4100bc0 2024-03-26T21:57:33.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.029+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81c40623e0 con 0x7f81c4100bc0 2024-03-26T21:57:33.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.030+0000 7f81c1ffb700 1 -- 172.21.15.196:0/3840961555 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f81bc01a3b0 con 0x7f81c4100bc0 2024-03-26T21:57:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:33.027 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:33.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81b0038810 msgr2=0x7f81b003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:33.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81b0038810 0x7f81b003acd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81b4006fd0 tx=0x7f81b4006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:33.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 msgr2=0x7f81c40ff220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:33.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c40ff220 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0062a0 tx=0x7f81bc00bd40 comp rx=0 tx=0).stop 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 shutdown_connections 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81b0038810 0x7f81b003acd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 --2- 172.21.15.196:0/3840961555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81c4100bc0 0x7f81c40ff220 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 >> 172.21.15.196:0/3840961555 conn(0x7f81c40fa370 msgr2=0x7f81c40fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 shutdown_connections 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:33.032+0000 7f81cac4b700 1 -- 172.21.15.196:0/3840961555 wait complete. 2024-03-26T21:57:33.030 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:34.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:33 smithi194 ceph-mon[29693]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:34.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:33 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3840961555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:34.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:34.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:34.833 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:35.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:35 smithi194 ceph-mon[29693]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:36.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.260+0000 7fec74abd700 1 -- 172.21.15.196:0/604226195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 msgr2=0x7fec700ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:36.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.260+0000 7fec74abd700 1 --2- 172.21.15.196:0/604226195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec700ffa40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec64009910 tx=0x7fec64009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:36.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.260+0000 7fec74abd700 1 -- 172.21.15.196:0/604226195 shutdown_connections 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.260+0000 7fec74abd700 1 --2- 172.21.15.196:0/604226195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec700ffa40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.260+0000 7fec74abd700 1 -- 172.21.15.196:0/604226195 >> 172.21.15.196:0/604226195 conn(0x7fec700fab80 msgr2=0x7fec700fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.261+0000 7fec74abd700 1 -- 172.21.15.196:0/604226195 shutdown_connections 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.261+0000 7fec74abd700 1 -- 172.21.15.196:0/604226195 wait complete. 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.261+0000 7fec74abd700 1 Processor -- start 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.261+0000 7fec74abd700 1 -- start start 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.261+0000 7fec74abd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec70193a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:36.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.261+0000 7fec74abd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec70193f50 con 0x7fec700ff620 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec6e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec70193a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec6e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec70193a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58002/0 (socket says 172.21.15.196:58002) 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec6e59c700 1 -- 172.21.15.196:0/371900069 learned_addr learned my addr 172.21.15.196:0/371900069 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec6e59c700 1 -- 172.21.15.196:0/371900069 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640095f0 con 0x7fec700ff620 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec6e59c700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec70193a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fec64006010 tx=0x7fec64009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec6401d070 con 0x7fec700ff620 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec70194150 con 0x7fec700ff620 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec64005dc0 con 0x7fec700ff620 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec64022e90 con 0x7fec700ff620 2024-03-26T21:57:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.262+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec701945f0 con 0x7fec700ff620 2024-03-26T21:57:36.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.263+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7018d340 con 0x7fec700ff620 2024-03-26T21:57:36.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.263+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fec6400b960 con 0x7fec700ff620 2024-03-26T21:57:36.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.263+0000 7fec5f7fe700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fec5803aa00 0x7fec5803cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:36.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.263+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec6404c730 con 0x7fec700ff620 2024-03-26T21:57:36.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.264+0000 7fec6dd9b700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fec5803aa00 0x7fec5803cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:36.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.264+0000 7fec6dd9b700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fec5803aa00 0x7fec5803cec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec60006fd0 tx=0x7fec60006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:36.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.269+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fec6400bd50 con 0x7fec700ff620 2024-03-26T21:57:36.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.540+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec700623e0 con 0x7fec700ff620 2024-03-26T21:57:36.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.540+0000 7fec5f7fe700 1 -- 172.21.15.196:0/371900069 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fec700623e0 con 0x7fec700ff620 2024-03-26T21:57:36.538 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:36.538 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:36.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fec5803aa00 msgr2=0x7fec5803cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:36.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fec5803aa00 0x7fec5803cec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec60006fd0 tx=0x7fec60006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:36.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 msgr2=0x7fec70193a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:36.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec70193a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fec64006010 tx=0x7fec64009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:36.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 shutdown_connections 2024-03-26T21:57:36.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fec5803aa00 0x7fec5803cec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:36.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 --2- 172.21.15.196:0/371900069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec700ff620 0x7fec70193a10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:36.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 >> 172.21.15.196:0/371900069 conn(0x7fec700fab80 msgr2=0x7fec700fb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:36.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 shutdown_connections 2024-03-26T21:57:36.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:36.543+0000 7fec74abd700 1 -- 172.21.15.196:0/371900069 wait complete. 2024-03-26T21:57:36.541 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:36.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:36 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/371900069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:37.912 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:37 smithi194 ceph-mon[29693]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:38.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:38.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:38.320 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:38.366 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:39.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 -- 172.21.15.196:0/561505153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 msgr2=0x7fb20c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:39.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 --2- 172.21.15.196:0/561505153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c100b40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009910 tx=0x7fb1fc009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:39.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 -- 172.21.15.196:0/561505153 shutdown_connections 2024-03-26T21:57:39.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 --2- 172.21.15.196:0/561505153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c100b40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 -- 172.21.15.196:0/561505153 >> 172.21.15.196:0/561505153 conn(0x7fb20c0fbcc0 msgr2=0x7fb20c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 -- 172.21.15.196:0/561505153 shutdown_connections 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 -- 172.21.15.196:0/561505153 wait complete. 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.896+0000 7fb211d23700 1 Processor -- start 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb211d23700 1 -- start start 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb211d23700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb211d23700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c196100 con 0x7fb20c100720 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb20b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb20b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58028/0 (socket says 172.21.15.196:58028) 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb20b7fe700 1 -- 172.21.15.196:0/1101883521 learned_addr learned my addr 172.21.15.196:0/1101883521 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:39.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.897+0000 7fb20b7fe700 1 -- 172.21.15.196:0/1101883521 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0095f0 con 0x7fb20c100720 2024-03-26T21:57:39.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.898+0000 7fb20b7fe700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c195bc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc006010 tx=0x7fb1fc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:39.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.898+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1fc01d070 con 0x7fb20c100720 2024-03-26T21:57:39.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.898+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c196300 con 0x7fb20c100720 2024-03-26T21:57:39.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.898+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1fc005dc0 con 0x7fb20c100720 2024-03-26T21:57:39.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.898+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1fc022e90 con 0x7fb20c100720 2024-03-26T21:57:39.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.898+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c1967a0 con 0x7fb20c100720 2024-03-26T21:57:39.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.899+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c18f550 con 0x7fb20c100720 2024-03-26T21:57:39.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.899+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb1fc022470 con 0x7fb20c100720 2024-03-26T21:57:39.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.899+0000 7fb208ff9700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1f4038760 0x7fb1f403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:39.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.899+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb1fc04c9e0 con 0x7fb20c100720 2024-03-26T21:57:39.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.899+0000 7fb20affd700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1f4038760 0x7fb1f403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:39.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.900+0000 7fb20affd700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1f4038760 0x7fb1f403ac20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb200006fd0 tx=0x7fb200006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:39.901 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:39.904+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1fc00b960 con 0x7fb20c100720 2024-03-26T21:57:40.088 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:39 smithi194 ceph-mon[29693]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:40.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.171+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb20c0623e0 con 0x7fb20c100720 2024-03-26T21:57:40.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.172+0000 7fb208ff9700 1 -- 172.21.15.196:0/1101883521 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb20c0623e0 con 0x7fb20c100720 2024-03-26T21:57:40.169 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:40.169 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:40.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1f4038760 msgr2=0x7fb1f403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:40.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1f4038760 0x7fb1f403ac20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb200006fd0 tx=0x7fb200006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:40.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 msgr2=0x7fb20c195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:40.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c195bc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc006010 tx=0x7fb1fc009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:40.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 shutdown_connections 2024-03-26T21:57:40.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1f4038760 0x7fb1f403ac20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:40.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 --2- 172.21.15.196:0/1101883521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb20c100720 0x7fb20c195bc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:40.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 >> 172.21.15.196:0/1101883521 conn(0x7fb20c0fbcc0 msgr2=0x7fb20c0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:40.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 shutdown_connections 2024-03-26T21:57:40.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:40.174+0000 7fb211d23700 1 -- 172.21.15.196:0/1101883521 wait complete. 2024-03-26T21:57:40.172 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:41.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:40 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1101883521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:41.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:41.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:41.943 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:41.988 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:42.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:41 smithi194 ceph-mon[29693]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:43.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 -- 172.21.15.196:0/4181225291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 msgr2=0x7f1454100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:43.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 --2- 172.21.15.196:0/4181225291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454100b40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1448009910 tx=0x7f1448009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:43.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 -- 172.21.15.196:0/4181225291 shutdown_connections 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 --2- 172.21.15.196:0/4181225291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454100b40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 -- 172.21.15.196:0/4181225291 >> 172.21.15.196:0/4181225291 conn(0x7f14540fbcc0 msgr2=0x7f14540fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 -- 172.21.15.196:0/4181225291 shutdown_connections 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 -- 172.21.15.196:0/4181225291 wait complete. 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 Processor -- start 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.538+0000 7f145ac15700 1 -- start start 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f145ac15700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f145ac15700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454074f60 con 0x7f1454100720 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f14589b1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f14589b1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:48448/0 (socket says 172.21.15.196:48448) 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f14589b1700 1 -- 172.21.15.196:0/4051956576 learned_addr learned my addr 172.21.15.196:0/4051956576 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f14589b1700 1 -- 172.21.15.196:0/4051956576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480095f0 con 0x7f1454100720 2024-03-26T21:57:43.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f14589b1700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454074a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1448006010 tx=0x7f1448009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.539+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f144801d070 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.540+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14540731a0 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.540+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1448005dc0 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.540+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1448022e90 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.540+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1454073640 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.541+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f144800f4f0 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.541+0000 7f1451ffb700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1444038760 0x7f144403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.541+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1454106e80 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.541+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f144804c920 con 0x7f1454100720 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.541+0000 7f1453fff700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1444038760 0x7f144403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:43.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.542+0000 7f1453fff700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1444038760 0x7f144403ac20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1440006fd0 tx=0x7f1440006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:43.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.546+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f144802a950 con 0x7f1454100720 2024-03-26T21:57:43.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.820+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14540623e0 con 0x7f1454100720 2024-03-26T21:57:43.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.821+0000 7f1451ffb700 1 -- 172.21.15.196:0/4051956576 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1448027020 con 0x7f1454100720 2024-03-26T21:57:43.818 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:43.818 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1444038760 msgr2=0x7f144403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1444038760 0x7f144403ac20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1440006fd0 tx=0x7f1440006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 msgr2=0x7f1454074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454074a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1448006010 tx=0x7f1448009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 shutdown_connections 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1444038760 0x7f144403ac20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 --2- 172.21.15.196:0/4051956576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1454100720 0x7f1454074a20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 >> 172.21.15.196:0/4051956576 conn(0x7f14540fbcc0 msgr2=0x7f14540fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 shutdown_connections 2024-03-26T21:57:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:43.823+0000 7f145ac15700 1 -- 172.21.15.196:0/4051956576 wait complete. 2024-03-26T21:57:43.822 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:44.166 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:43 smithi194 ceph-mon[29693]: pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:45.095 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:44 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4051956576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:45.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:45.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:45.606 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:45.651 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:46.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:45 smithi194 ceph-mon[29693]: pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:47.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/3101818650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 msgr2=0x7fd9c0100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:47.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 --2- 172.21.15.196:0/3101818650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0100b60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000b0a0 tx=0x7fd9b000b3b0 comp rx=0 tx=0).stop 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/3101818650 shutdown_connections 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 --2- 172.21.15.196:0/3101818650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0100b60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/3101818650 >> 172.21.15.196:0/3101818650 conn(0x7fd9c00fbd00 msgr2=0x7fd9c00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/3101818650 shutdown_connections 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.218+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/3101818650 wait complete. 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9c5ff8700 1 Processor -- start 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9c5ff8700 1 -- start start 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9c5ff8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9c5ff8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c01960d0 con 0x7fd9c0100740 2024-03-26T21:57:47.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9bf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9bf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:48464/0 (socket says 172.21.15.196:48464) 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.219+0000 7fd9bf7fe700 1 -- 172.21.15.196:0/288999216 learned_addr learned my addr 172.21.15.196:0/288999216 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9bf7fe700 1 -- 172.21.15.196:0/288999216 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b0009d00 con 0x7fd9c0100740 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9bf7fe700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0195b90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000bc90 tx=0x7fd9b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9b000e040 con 0x7fd9c0100740 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c01962d0 con 0x7fd9c0100740 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9b0003d50 con 0x7fd9c0100740 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c0196770 con 0x7fd9c0100740 2024-03-26T21:57:47.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.220+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9b001c8e0 con 0x7fd9c0100740 2024-03-26T21:57:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.221+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fd9b001ca40 con 0x7fd9c0100740 2024-03-26T21:57:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.221+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c018f570 con 0x7fd9c0100740 2024-03-26T21:57:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.221+0000 7fd9bcff9700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9a8038810 0x7fd9a803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.221+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd9b004c990 con 0x7fd9c0100740 2024-03-26T21:57:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.221+0000 7fd9beffd700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9a8038810 0x7fd9a803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.222+0000 7fd9beffd700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9a8038810 0x7fd9a803acd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4006fd0 tx=0x7fd9b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:47.223 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.227+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd9b001c400 con 0x7fd9c0100740 2024-03-26T21:57:47.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.491+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9c00623e0 con 0x7fd9c0100740 2024-03-26T21:57:47.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.492+0000 7fd9bcff9700 1 -- 172.21.15.196:0/288999216 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd9b001a3b0 con 0x7fd9c0100740 2024-03-26T21:57:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:47.489 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:47.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9a8038810 msgr2=0x7fd9a803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:47.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9a8038810 0x7fd9a803acd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4006fd0 tx=0x7fd9b4006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 msgr2=0x7fd9c0195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0195b90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000bc90 tx=0x7fd9b000bd70 comp rx=0 tx=0).stop 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 shutdown_connections 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9a8038810 0x7fd9a803acd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 --2- 172.21.15.196:0/288999216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9c0100740 0x7fd9c0195b90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 >> 172.21.15.196:0/288999216 conn(0x7fd9c00fbd00 msgr2=0x7fd9c00fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 shutdown_connections 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:47.494+0000 7fd9c5ff8700 1 -- 172.21.15.196:0/288999216 wait complete. 2024-03-26T21:57:47.492 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:48.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:47 smithi194 ceph-mon[29693]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:48.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:47 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/288999216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:49.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:49.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:49.322 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:50.102 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:49 smithi194 ceph-mon[29693]: pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:50.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 -- 172.21.15.196:0/2252739845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 msgr2=0x7f0f0c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 --2- 172.21.15.196:0/2252739845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c100b60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400b0a0 tx=0x7f0f0400b3b0 comp rx=0 tx=0).stop 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 -- 172.21.15.196:0/2252739845 shutdown_connections 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 --2- 172.21.15.196:0/2252739845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c100b60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 -- 172.21.15.196:0/2252739845 >> 172.21.15.196:0/2252739845 conn(0x7f0f0c0fbd00 msgr2=0x7f0f0c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 -- 172.21.15.196:0/2252739845 shutdown_connections 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.914+0000 7f0f10f06700 1 -- 172.21.15.196:0/2252739845 wait complete. 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.915+0000 7f0f10f06700 1 Processor -- start 2024-03-26T21:57:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.915+0000 7f0f10f06700 1 -- start start 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.915+0000 7f0f10f06700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.915+0000 7f0f10f06700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c196100 con 0x7f0f0c100740 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f0a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f0a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:48490/0 (socket says 172.21.15.196:48490) 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f0a59c700 1 -- 172.21.15.196:0/4014038384 learned_addr learned my addr 172.21.15.196:0/4014038384 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f0a59c700 1 -- 172.21.15.196:0/4014038384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04009d00 con 0x7f0f0c100740 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f0a59c700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c195bc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0f040062a0 tx=0x7f0f0400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f0400e040 con 0x7f0f0c100740 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f04003d70 con 0x7f0f0c100740 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c196300 con 0x7f0f0c100740 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f0401c890 con 0x7f0f0c100740 2024-03-26T21:57:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.916+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c1967a0 con 0x7f0f0c100740 2024-03-26T21:57:50.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.917+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c18f570 con 0x7f0f0c100740 2024-03-26T21:57:50.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.917+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f0f0401cb50 con 0x7f0f0c100740 2024-03-26T21:57:50.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.917+0000 7f0efb7fe700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0ef40387c0 0x7f0ef403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:50.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.917+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0f0404c9a0 con 0x7f0f0c100740 2024-03-26T21:57:50.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.918+0000 7f0f09d9b700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0ef40387c0 0x7f0ef403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:50.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.918+0000 7f0f09d9b700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0ef40387c0 0x7f0ef403ac80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f00006fd0 tx=0x7f0f00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:50.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:50.923+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f0401a3b0 con 0x7f0f0c100740 2024-03-26T21:57:51.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.192+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f0c0623e0 con 0x7f0f0c100740 2024-03-26T21:57:51.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.192+0000 7f0efb7fe700 1 -- 172.21.15.196:0/4014038384 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f0f0c0623e0 con 0x7f0f0c100740 2024-03-26T21:57:51.190 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:51.190 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0ef40387c0 msgr2=0x7f0ef403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0ef40387c0 0x7f0ef403ac80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f00006fd0 tx=0x7f0f00006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 msgr2=0x7f0f0c195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c195bc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0f040062a0 tx=0x7f0f0400bd40 comp rx=0 tx=0).stop 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 shutdown_connections 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0ef40387c0 0x7f0ef403ac80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 --2- 172.21.15.196:0/4014038384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f0c100740 0x7f0f0c195bc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 >> 172.21.15.196:0/4014038384 conn(0x7f0f0c0fbd00 msgr2=0x7f0f0c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:51.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.194+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 shutdown_connections 2024-03-26T21:57:51.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:51.195+0000 7f0f10f06700 1 -- 172.21.15.196:0/4014038384 wait complete. 2024-03-26T21:57:51.193 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:52.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:51 smithi194 ceph-mon[29693]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:52.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:51 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4014038384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:52.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:52.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:52.971 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:53.018 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:54.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:53 smithi194 ceph-mon[29693]: pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:54.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 -- 172.21.15.196:0/3673084354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 msgr2=0x7f97f0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:54.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 --2- 172.21.15.196:0/3673084354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0100b40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009910 tx=0x7f97e0009c20 comp rx=0 tx=0).stop 2024-03-26T21:57:54.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 -- 172.21.15.196:0/3673084354 shutdown_connections 2024-03-26T21:57:54.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 --2- 172.21.15.196:0/3673084354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0100b40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:54.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 -- 172.21.15.196:0/3673084354 >> 172.21.15.196:0/3673084354 conn(0x7f97f00fbcc0 msgr2=0x7f97f00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:54.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 -- 172.21.15.196:0/3673084354 shutdown_connections 2024-03-26T21:57:54.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.457+0000 7f97f6399700 1 -- 172.21.15.196:0/3673084354 wait complete. 2024-03-26T21:57:54.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97f6399700 1 Processor -- start 2024-03-26T21:57:54.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97f6399700 1 -- start start 2024-03-26T21:57:54.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97f6399700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97f6399700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0196100 con 0x7f97f0100720 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97effff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97effff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49846/0 (socket says 172.21.15.196:49846) 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.458+0000 7f97effff700 1 -- 172.21.15.196:0/3669755149 learned_addr learned my addr 172.21.15.196:0/3669755149 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97effff700 1 -- 172.21.15.196:0/3669755149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00095f0 con 0x7f97f0100720 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97effff700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0195bc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006010 tx=0x7f97e0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97e001d070 con 0x7f97f0100720 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97e0005dc0 con 0x7f97f0100720 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f0196300 con 0x7f97f0100720 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97e0022e90 con 0x7f97f0100720 2024-03-26T21:57:54.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.459+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f01967a0 con 0x7f97f0100720 2024-03-26T21:57:54.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.460+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f018f550 con 0x7f97f0100720 2024-03-26T21:57:54.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.460+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f97e000b960 con 0x7f97f0100720 2024-03-26T21:57:54.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.460+0000 7f97ed7fa700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d8038760 0x7f97d803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:54.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.460+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f97e004c8c0 con 0x7f97f0100720 2024-03-26T21:57:54.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.460+0000 7f97ef7fe700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d8038760 0x7f97d803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:54.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.461+0000 7f97ef7fe700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d8038760 0x7f97d803ac20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97e4006fd0 tx=0x7f97e4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:54.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.465+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97e000f4f0 con 0x7f97f0100720 2024-03-26T21:57:54.737 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.739+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97f00623e0 con 0x7f97f0100720 2024-03-26T21:57:54.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.740+0000 7f97ed7fa700 1 -- 172.21.15.196:0/3669755149 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f97e0027030 con 0x7f97f0100720 2024-03-26T21:57:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:54.738 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d8038760 msgr2=0x7f97d803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d8038760 0x7f97d803ac20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97e4006fd0 tx=0x7f97e4006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 msgr2=0x7f97f0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0195bc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006010 tx=0x7f97e0009f80 comp rx=0 tx=0).stop 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 shutdown_connections 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f97d8038760 0x7f97d803ac20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 --2- 172.21.15.196:0/3669755149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97f0100720 0x7f97f0195bc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 >> 172.21.15.196:0/3669755149 conn(0x7f97f00fbcc0 msgr2=0x7f97f00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 shutdown_connections 2024-03-26T21:57:54.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:54.742+0000 7f97f6399700 1 -- 172.21.15.196:0/3669755149 wait complete. 2024-03-26T21:57:54.741 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:55.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:55 smithi194 ceph-mon[29693]: pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:55.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:55 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3669755149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:56.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:56.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:56.525 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:56.569 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:57:57.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 -- 172.21.15.196:0/3331105685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 msgr2=0x7f20f0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:57.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 --2- 172.21.15.196:0/3331105685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0100b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20e800b0a0 tx=0x7f20e800b3b0 comp rx=0 tx=0).stop 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 -- 172.21.15.196:0/3331105685 shutdown_connections 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 --2- 172.21.15.196:0/3331105685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0100b40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 -- 172.21.15.196:0/3331105685 >> 172.21.15.196:0/3331105685 conn(0x7f20f00fbcc0 msgr2=0x7f20f00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 -- 172.21.15.196:0/3331105685 shutdown_connections 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.911+0000 7f20f7c1e700 1 -- 172.21.15.196:0/3331105685 wait complete. 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.912+0000 7f20f7c1e700 1 Processor -- start 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.912+0000 7f20f7c1e700 1 -- start start 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.912+0000 7f20f7c1e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.912+0000 7f20f7c1e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f0196100 con 0x7f20f0100720 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.912+0000 7f20f59ba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.912+0000 7f20f59ba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49872/0 (socket says 172.21.15.196:49872) 2024-03-26T21:57:57.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20f59ba700 1 -- 172.21.15.196:0/671878918 learned_addr learned my addr 172.21.15.196:0/671878918 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20f59ba700 1 -- 172.21.15.196:0/671878918 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e8009d00 con 0x7f20f0100720 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20f59ba700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0195bc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f20e80062a0 tx=0x7f20e800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20e800e040 con 0x7f20f0100720 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20e8003d70 con 0x7f20f0100720 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20e801c890 con 0x7f20f0100720 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f0196300 con 0x7f20f0100720 2024-03-26T21:57:57.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.913+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f01967a0 con 0x7f20f0100720 2024-03-26T21:57:57.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.914+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f018f550 con 0x7f20f0100720 2024-03-26T21:57:57.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.914+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f20e801cb50 con 0x7f20f0100720 2024-03-26T21:57:57.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.914+0000 7f20e6ffd700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f20dc0387c0 0x7f20dc03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:57:57.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.914+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f20e804ca60 con 0x7f20f0100720 2024-03-26T21:57:57.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.914+0000 7f20f51b9700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f20dc0387c0 0x7f20dc03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:57:57.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.915+0000 7f20f51b9700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f20dc0387c0 0x7f20dc03ac80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20e0006fd0 tx=0x7f20e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:57:57.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:57.919+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f20e801a3b0 con 0x7f20f0100720 2024-03-26T21:57:58.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:57 smithi194 ceph-mon[29693]: pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:57:58.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.190+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20f00623e0 con 0x7f20f0100720 2024-03-26T21:57:58.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.190+0000 7f20e6ffd700 1 -- 172.21.15.196:0/671878918 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f20f00623e0 con 0x7f20f0100720 2024-03-26T21:57:58.188 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:57:58.188 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:57:58.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f20dc0387c0 msgr2=0x7f20dc03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:58.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f20dc0387c0 0x7f20dc03ac80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20e0006fd0 tx=0x7f20e0006e40 comp rx=0 tx=0).stop 2024-03-26T21:57:58.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 msgr2=0x7f20f0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:58.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0195bc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f20e80062a0 tx=0x7f20e800bd40 comp rx=0 tx=0).stop 2024-03-26T21:57:58.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 shutdown_connections 2024-03-26T21:57:58.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f20dc0387c0 0x7f20dc03ac80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:58.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 --2- 172.21.15.196:0/671878918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20f0100720 0x7f20f0195bc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:57:58.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 >> 172.21.15.196:0/671878918 conn(0x7f20f00fbcc0 msgr2=0x7f20f00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:58.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.192+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 shutdown_connections 2024-03-26T21:57:58.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:57:58.193+0000 7f20f7c1e700 1 -- 172.21.15.196:0/671878918 wait complete. 2024-03-26T21:57:58.191 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:57:59.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:58 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/671878918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:57:59.754 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:57:59.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:57:59.958 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:00.002 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:00.114 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:57:59 smithi194 ceph-mon[29693]: pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1178834663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 msgr2=0x7f2b94100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 --2- 172.21.15.196:0/1178834663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94100b40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009910 tx=0x7f2b84009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1178834663 shutdown_connections 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 --2- 172.21.15.196:0/1178834663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94100b40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1178834663 >> 172.21.15.196:0/1178834663 conn(0x7f2b940fbcc0 msgr2=0x7f2b940fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1178834663 shutdown_connections 2024-03-26T21:58:01.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.476+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1178834663 wait complete. 2024-03-26T21:58:01.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b98ea5700 1 Processor -- start 2024-03-26T21:58:01.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b98ea5700 1 -- start start 2024-03-26T21:58:01.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b98ea5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:01.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b98ea5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b94196100 con 0x7f2b94100720 2024-03-26T21:58:01.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b9259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b9259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49886/0 (socket says 172.21.15.196:49886) 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.477+0000 7f2b9259c700 1 -- 172.21.15.196:0/1480879125 learned_addr learned my addr 172.21.15.196:0/1480879125 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b9259c700 1 -- 172.21.15.196:0/1480879125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b840095f0 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b9259c700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94195bc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006010 tx=0x7f2b84009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b8401d070 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b94196300 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b941967a0 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b84005dc0 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.478+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b84022e70 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.479+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b74005320 con 0x7f2b94100720 2024-03-26T21:58:01.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.479+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2b8400b960 con 0x7f2b94100720 2024-03-26T21:58:01.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.479+0000 7f2b8bfff700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2b80038770 0x7f2b8003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:01.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.479+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2b8404c760 con 0x7f2b94100720 2024-03-26T21:58:01.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.480+0000 7f2b89d9b700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2b80038770 0x7f2b8003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:01.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.480+0000 7f2b89d9b700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2b80038770 0x7f2b8003ac30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c006fd0 tx=0x7f2b7c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:01.482 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.485+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2b8400f4b0 con 0x7f2b94100720 2024-03-26T21:58:01.745 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.747+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b74005190 con 0x7f2b94100720 2024-03-26T21:58:01.745 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.748+0000 7f2b8bfff700 1 -- 172.21.15.196:0/1480879125 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2b84027030 con 0x7f2b94100720 2024-03-26T21:58:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:01.746 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2b80038770 msgr2=0x7f2b8003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2b80038770 0x7f2b8003ac30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c006fd0 tx=0x7f2b7c006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 msgr2=0x7f2b94195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94195bc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006010 tx=0x7f2b84009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 shutdown_connections 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2b80038770 0x7f2b8003ac30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 --2- 172.21.15.196:0/1480879125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b94100720 0x7f2b94195bc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 >> 172.21.15.196:0/1480879125 conn(0x7f2b940fbcc0 msgr2=0x7f2b940fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 shutdown_connections 2024-03-26T21:58:01.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:01.751+0000 7f2b98ea5700 1 -- 172.21.15.196:0/1480879125 wait complete. 2024-03-26T21:58:01.750 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:02.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:01 smithi194 ceph-mon[29693]: pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:03.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:02 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1480879125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:03.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:03.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:03.495 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:03.548 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:04.162 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:03 smithi194 ceph-mon[29693]: pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3995342829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 msgr2=0x7f7b980ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 --2- 172.21.15.196:0/3995342829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980ffa70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009910 tx=0x7f7b94009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3995342829 shutdown_connections 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 --2- 172.21.15.196:0/3995342829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980ffa70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3995342829 >> 172.21.15.196:0/3995342829 conn(0x7f7b980fac10 msgr2=0x7f7b980fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3995342829 shutdown_connections 2024-03-26T21:58:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.118+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3995342829 wait complete. 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.119+0000 7f7b9fa56700 1 Processor -- start 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.119+0000 7f7b9fa56700 1 -- start start 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.119+0000 7f7b9fa56700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.119+0000 7f7b9fa56700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98069a10 con 0x7f7b980ff650 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.119+0000 7f7b9d7f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b9d7f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44522/0 (socket says 172.21.15.196:44522) 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b9d7f2700 1 -- 172.21.15.196:0/3284237487 learned_addr learned my addr 172.21.15.196:0/3284237487 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b9d7f2700 1 -- 172.21.15.196:0/3284237487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940095f0 con 0x7f7b980ff650 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b9d7f2700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980694d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b94006010 tx=0x7f7b94009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:05.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b9401d070 con 0x7f7b980ff650 2024-03-26T21:58:05.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b981966d0 con 0x7f7b980ff650 2024-03-26T21:58:05.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b94005dc0 con 0x7f7b980ff650 2024-03-26T21:58:05.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b94022e90 con 0x7f7b980ff650 2024-03-26T21:58:05.119 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.120+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b98196b70 con 0x7f7b980ff650 2024-03-26T21:58:05.120 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.121+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f7b94022470 con 0x7f7b980ff650 2024-03-26T21:58:05.120 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.121+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9818e480 con 0x7f7b980ff650 2024-03-26T21:58:05.120 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.121+0000 7f7b8e7fc700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7b84038760 0x7f7b8403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:05.120 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.121+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7b9404c7f0 con 0x7f7b980ff650 2024-03-26T21:58:05.120 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.122+0000 7f7b9cff1700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7b84038760 0x7f7b8403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:05.120 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.122+0000 7f7b9cff1700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7b84038760 0x7f7b8403ac20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006fd0 tx=0x7f7b88006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:05.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.127+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b9400b960 con 0x7f7b980ff650 2024-03-26T21:58:05.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.395+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b980623e0 con 0x7f7b980ff650 2024-03-26T21:58:05.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.395+0000 7f7b8e7fc700 1 -- 172.21.15.196:0/3284237487 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7b980623e0 con 0x7f7b980ff650 2024-03-26T21:58:05.393 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:05.393 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:05.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7b84038760 msgr2=0x7f7b8403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:05.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7b84038760 0x7f7b8403ac20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006fd0 tx=0x7f7b88006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:05.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 msgr2=0x7f7b980694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:05.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980694d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b94006010 tx=0x7f7b94009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:05.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 shutdown_connections 2024-03-26T21:58:05.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7b84038760 0x7f7b8403ac20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:05.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 --2- 172.21.15.196:0/3284237487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b980ff650 0x7f7b980694d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:05.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 >> 172.21.15.196:0/3284237487 conn(0x7f7b980fac10 msgr2=0x7f7b980fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:05.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 shutdown_connections 2024-03-26T21:58:05.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:05.397+0000 7f7b9fa56700 1 -- 172.21.15.196:0/3284237487 wait complete. 2024-03-26T21:58:05.396 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:05.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:05 smithi194 ceph-mon[29693]: pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:06.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:06 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3284237487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:07.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:07.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:07.206 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:07.252 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:07.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:07 smithi194 ceph-mon[29693]: pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:08.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 -- 172.21.15.196:0/1255301951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 msgr2=0x7f6d001035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:08.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 --2- 172.21.15.196:0/1255301951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001035b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009910 tx=0x7f6cf4009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:08.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 -- 172.21.15.196:0/1255301951 shutdown_connections 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 --2- 172.21.15.196:0/1255301951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001035b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 -- 172.21.15.196:0/1255301951 >> 172.21.15.196:0/1255301951 conn(0x7f6d000fabd0 msgr2=0x7f6d000fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 -- 172.21.15.196:0/1255301951 shutdown_connections 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.704+0000 7f6d06de1700 1 -- 172.21.15.196:0/1255301951 wait complete. 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d06de1700 1 Processor -- start 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d06de1700 1 -- start start 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d06de1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d06de1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d00100f30 con 0x7f6d001011c0 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d04b7d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:08.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d04b7d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44552/0 (socket says 172.21.15.196:44552) 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d04b7d700 1 -- 172.21.15.196:0/2871101701 learned_addr learned my addr 172.21.15.196:0/2871101701 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d04b7d700 1 -- 172.21.15.196:0/2871101701 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40095f0 con 0x7f6d001011c0 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.705+0000 7f6d04b7d700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001009f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4005b40 tx=0x7f6cf4005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.706+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf401d070 con 0x7f6d001011c0 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.706+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d000ff170 con 0x7f6d001011c0 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.706+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cf4005dc0 con 0x7f6d001011c0 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.706+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf4022e00 con 0x7f6d001011c0 2024-03-26T21:58:08.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.706+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d000ff610 con 0x7f6d001011c0 2024-03-26T21:58:08.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.707+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d00106e90 con 0x7f6d001011c0 2024-03-26T21:58:08.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.707+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6cf400b960 con 0x7f6d001011c0 2024-03-26T21:58:08.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.707+0000 7f6cfdffb700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:08.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.707+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6cf404c8f0 con 0x7f6d001011c0 2024-03-26T21:58:08.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.707+0000 7f6cfffff700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:08.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.707+0000 7f6cfffff700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006fd0 tx=0x7f6cec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:08.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.712+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6cf400f4b0 con 0x7f6d001011c0 2024-03-26T21:58:08.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.979+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d0018ef40 con 0x7f6d001011c0 2024-03-26T21:58:08.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.980+0000 7f6cfdffb700 1 -- 172.21.15.196:0/2871101701 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6cf4027070 con 0x7f6d001011c0 2024-03-26T21:58:08.978 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:08.978 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 msgr2=0x7f6cf003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006fd0 tx=0x7f6cec006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 msgr2=0x7f6d001009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001009f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4005b40 tx=0x7f6cf4005000 comp rx=0 tx=0).stop 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 shutdown_connections 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 --2- 172.21.15.196:0/2871101701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d001011c0 0x7f6d001009f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 >> 172.21.15.196:0/2871101701 conn(0x7f6d000fabd0 msgr2=0x7f6d000fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 shutdown_connections 2024-03-26T21:58:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:08.982+0000 7f6d06de1700 1 -- 172.21.15.196:0/2871101701 wait complete. 2024-03-26T21:58:08.981 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:10.116 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:09 smithi194 ceph-mon[29693]: pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:10.116 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:09 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2871101701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:10.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:10.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:10.744 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:10.791 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:12.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:11 smithi194 ceph-mon[29693]: pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 -- 172.21.15.196:0/1318818141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 msgr2=0x7f66a8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 --2- 172.21.15.196:0/1318818141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8100b40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009910 tx=0x7f66a0009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 -- 172.21.15.196:0/1318818141 shutdown_connections 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 --2- 172.21.15.196:0/1318818141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8100b40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 -- 172.21.15.196:0/1318818141 >> 172.21.15.196:0/1318818141 conn(0x7f66a80fbcc0 msgr2=0x7f66a80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 -- 172.21.15.196:0/1318818141 shutdown_connections 2024-03-26T21:58:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.282+0000 7f66af961700 1 -- 172.21.15.196:0/1318818141 wait complete. 2024-03-26T21:58:12.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66af961700 1 Processor -- start 2024-03-26T21:58:12.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66af961700 1 -- start start 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66af961700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66af961700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a8196100 con 0x7f66a8100720 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66ad6fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66ad6fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44568/0 (socket says 172.21.15.196:44568) 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66ad6fd700 1 -- 172.21.15.196:0/3940025891 learned_addr learned my addr 172.21.15.196:0/3940025891 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.283+0000 7f66ad6fd700 1 -- 172.21.15.196:0/3940025891 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00095f0 con 0x7f66a8100720 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.284+0000 7f66ad6fd700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8195bc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f66a0006010 tx=0x7f66a00050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.284+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a001d070 con 0x7f66a8100720 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.284+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66a000be20 con 0x7f66a8100720 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.284+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a8196300 con 0x7f66a8100720 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.284+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a0022e50 con 0x7f66a8100720 2024-03-26T21:58:12.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.284+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a81967a0 con 0x7f66a8100720 2024-03-26T21:58:12.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.285+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a818f550 con 0x7f66a8100720 2024-03-26T21:58:12.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.285+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f66a0022700 con 0x7f66a8100720 2024-03-26T21:58:12.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.285+0000 7f669e7fc700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6694038760 0x7f669403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:12.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.285+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f66a004cc30 con 0x7f66a8100720 2024-03-26T21:58:12.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.285+0000 7f66acefc700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6694038760 0x7f669403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:12.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.286+0000 7f66acefc700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6694038760 0x7f669403ac20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6698006fd0 tx=0x7f6698006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:12.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.291+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f66a000b960 con 0x7f66a8100720 2024-03-26T21:58:12.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.556+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66a80623e0 con 0x7f66a8100720 2024-03-26T21:58:12.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.556+0000 7f669e7fc700 1 -- 172.21.15.196:0/3940025891 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f66a80623e0 con 0x7f66a8100720 2024-03-26T21:58:12.554 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:12.554 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.558+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6694038760 msgr2=0x7f669403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.558+0000 7f66af961700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6694038760 0x7f669403ac20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6698006fd0 tx=0x7f6698006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 msgr2=0x7f66a8195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8195bc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f66a0006010 tx=0x7f66a00050b0 comp rx=0 tx=0).stop 2024-03-26T21:58:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 shutdown_connections 2024-03-26T21:58:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6694038760 0x7f669403ac20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 --2- 172.21.15.196:0/3940025891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66a8100720 0x7f66a8195bc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 >> 172.21.15.196:0/3940025891 conn(0x7f66a80fbcc0 msgr2=0x7f66a80fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 shutdown_connections 2024-03-26T21:58:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:12.559+0000 7f66af961700 1 -- 172.21.15.196:0/3940025891 wait complete. 2024-03-26T21:58:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:13.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:12 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3940025891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:14.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:14.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:14.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:13 smithi194 ceph-mon[29693]: pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:14.332 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:14.376 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:15.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:15 smithi194 ceph-mon[29693]: pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:15.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3113582909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 msgr2=0x7f4a000ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:15.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 --2- 172.21.15.196:0/3113582909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a000ffa50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009910 tx=0x7f49f0009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:15.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3113582909 shutdown_connections 2024-03-26T21:58:15.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 --2- 172.21.15.196:0/3113582909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a000ffa50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3113582909 >> 172.21.15.196:0/3113582909 conn(0x7f4a000fabd0 msgr2=0x7f4a000fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3113582909 shutdown_connections 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3113582909 wait complete. 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.857+0000 7f4a05f4c700 1 Processor -- start 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f4a05f4c700 1 -- start start 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f4a05f4c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a00195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:15.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f4a05f4c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a001961c0 con 0x7f4a000ff630 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f49ff7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a00195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f49ff7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a00195c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:50616/0 (socket says 172.21.15.196:50616) 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f49ff7fe700 1 -- 172.21.15.196:0/3965664784 learned_addr learned my addr 172.21.15.196:0/3965664784 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.858+0000 7f49ff7fe700 1 -- 172.21.15.196:0/3965664784 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f00095f0 con 0x7f4a000ff630 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.859+0000 7f49ff7fe700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a00195c80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004f40 tx=0x7f49f0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.859+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49f001d070 con 0x7f4a000ff630 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.859+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a001963c0 con 0x7f4a000ff630 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.859+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49f0005dc0 con 0x7f4a000ff630 2024-03-26T21:58:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.859+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49f0022e40 con 0x7f4a000ff630 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.859+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a00196860 con 0x7f4a000ff630 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.860+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0018f780 con 0x7f4a000ff630 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.860+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f49f00226c0 con 0x7f4a000ff630 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.860+0000 7f49fcff9700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49e8038760 0x7f49e803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.860+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f49f004cb10 con 0x7f4a000ff630 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.860+0000 7f49feffd700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49e8038760 0x7f49e803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.861+0000 7f49feffd700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49e8038760 0x7f49e803ac20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f49f4006fd0 tx=0x7f49f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:15.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:15.865+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49f002ba00 con 0x7f4a000ff630 2024-03-26T21:58:16.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.137+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a0002ccb0 con 0x7f4a000ff630 2024-03-26T21:58:16.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.137+0000 7f49fcff9700 1 -- 172.21.15.196:0/3965664784 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f49f002b3b0 con 0x7f4a000ff630 2024-03-26T21:58:16.135 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:16.135 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49e8038760 msgr2=0x7f49e803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49e8038760 0x7f49e803ac20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f49f4006fd0 tx=0x7f49f4006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 msgr2=0x7f4a00195c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a00195c80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004f40 tx=0x7f49f0004740 comp rx=0 tx=0).stop 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 shutdown_connections 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49e8038760 0x7f49e803ac20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 --2- 172.21.15.196:0/3965664784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a000ff630 0x7f4a00195c80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.139+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 >> 172.21.15.196:0/3965664784 conn(0x7f4a000fabd0 msgr2=0x7f4a00068f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:16.138 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.140+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 shutdown_connections 2024-03-26T21:58:16.138 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:16.140+0000 7f4a05f4c700 1 -- 172.21.15.196:0/3965664784 wait complete. 2024-03-26T21:58:16.138 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:16.662 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:16 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3965664784' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:17.663 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:17 smithi194 ceph-mon[29693]: pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:17.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:17.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:17.924 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:17.970 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:19.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 -- 172.21.15.196:0/3709574635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 msgr2=0x7f4e80100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:19.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 --2- 172.21.15.196:0/3709574635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80100b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b0a0 tx=0x7f4e7800b3b0 comp rx=0 tx=0).stop 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 -- 172.21.15.196:0/3709574635 shutdown_connections 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 --2- 172.21.15.196:0/3709574635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80100b80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 -- 172.21.15.196:0/3709574635 >> 172.21.15.196:0/3709574635 conn(0x7f4e800fbd40 msgr2=0x7f4e800fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 -- 172.21.15.196:0/3709574635 shutdown_connections 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.469+0000 7f4e885f6700 1 -- 172.21.15.196:0/3709574635 wait complete. 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e885f6700 1 Processor -- start 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e885f6700 1 -- start start 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e885f6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e885f6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80196140 con 0x7f4e80100760 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e86392700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e86392700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:50630/0 (socket says 172.21.15.196:50630) 2024-03-26T21:58:19.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e86392700 1 -- 172.21.15.196:0/565381976 learned_addr learned my addr 172.21.15.196:0/565381976 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.470+0000 7f4e86392700 1 -- 172.21.15.196:0/565381976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78009d00 con 0x7f4e80100760 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.471+0000 7f4e86392700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80195c00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e780062a0 tx=0x7f4e7800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.471+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7800e040 con 0x7f4e80100760 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.471+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e80196340 con 0x7f4e80100760 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.471+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e801967e0 con 0x7f4e80100760 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.471+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e78003d70 con 0x7f4e80100760 2024-03-26T21:58:19.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.471+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e780268e0 con 0x7f4e80100760 2024-03-26T21:58:19.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.472+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f4e78012640 con 0x7f4e80100760 2024-03-26T21:58:19.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.472+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8018f590 con 0x7f4e80100760 2024-03-26T21:58:19.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.472+0000 7f4e777fe700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4e6c0387c0 0x7f4e6c03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:19.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.472+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e7804c8e0 con 0x7f4e80100760 2024-03-26T21:58:19.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.472+0000 7f4e85b91700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4e6c0387c0 0x7f4e6c03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:19.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.473+0000 7f4e85b91700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4e6c0387c0 0x7f4e6c03ac80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:19.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.478+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4e78024540 con 0x7f4e80100760 2024-03-26T21:58:19.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.748+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e800623e0 con 0x7f4e80100760 2024-03-26T21:58:19.746 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.749+0000 7f4e777fe700 1 -- 172.21.15.196:0/565381976 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4e800623e0 con 0x7f4e80100760 2024-03-26T21:58:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:19.747 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:19.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4e6c0387c0 msgr2=0x7f4e6c03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:19.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4e6c0387c0 0x7f4e6c03ac80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e70006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:19.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 msgr2=0x7f4e80195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80195c00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e780062a0 tx=0x7f4e7800bd40 comp rx=0 tx=0).stop 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 shutdown_connections 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4e6c0387c0 0x7f4e6c03ac80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 --2- 172.21.15.196:0/565381976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e80100760 0x7f4e80195c00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 >> 172.21.15.196:0/565381976 conn(0x7f4e800fbd40 msgr2=0x7f4e800fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 shutdown_connections 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:19.751+0000 7f4e885f6700 1 -- 172.21.15.196:0/565381976 wait complete. 2024-03-26T21:58:19.749 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:20.140 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:19 smithi194 ceph-mon[29693]: pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:21.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:20 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/565381976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:21.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:21.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:21.509 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:21.555 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:22.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:21 smithi194 ceph-mon[29693]: pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:23.115 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 -- 172.21.15.196:0/2893895621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 msgr2=0x7ff81c0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 --2- 172.21.15.196:0/2893895621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0ffa50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009910 tx=0x7ff80c009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 -- 172.21.15.196:0/2893895621 shutdown_connections 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 --2- 172.21.15.196:0/2893895621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0ffa50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 -- 172.21.15.196:0/2893895621 >> 172.21.15.196:0/2893895621 conn(0x7ff81c0fabd0 msgr2=0x7ff81c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 -- 172.21.15.196:0/2893895621 shutdown_connections 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.116+0000 7ff82242c700 1 -- 172.21.15.196:0/2893895621 wait complete. 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff82242c700 1 Processor -- start 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff82242c700 1 -- start start 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff82242c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff82242c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c0fcf30 con 0x7ff81c0ff630 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0fc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34360/0 (socket says 172.21.15.196:34360) 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.117+0000 7ff81bfff700 1 -- 172.21.15.196:0/3850099361 learned_addr learned my addr 172.21.15.196:0/3850099361 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff81bfff700 1 -- 172.21.15.196:0/3850099361 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0095f0 con 0x7ff81c0ff630 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff81bfff700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0fc9f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff80c006010 tx=0x7ff80c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff80c01d070 con 0x7ff81c0ff630 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c0fb1c0 con 0x7ff81c0ff630 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff80c005dc0 con 0x7ff81c0ff630 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff80c022e90 con 0x7ff81c0ff630 2024-03-26T21:58:23.117 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.118+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c0fb660 con 0x7ff81c0ff630 2024-03-26T21:58:23.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.119+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff80c022470 con 0x7ff81c0ff630 2024-03-26T21:58:23.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.119+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81c0fc070 con 0x7ff81c0ff630 2024-03-26T21:58:23.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.119+0000 7ff8197fa700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff804038760 0x7ff80403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:23.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.119+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff80c04c870 con 0x7ff81c0ff630 2024-03-26T21:58:23.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.119+0000 7ff81b7fe700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff804038760 0x7ff80403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:23.118 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.120+0000 7ff81b7fe700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff804038760 0x7ff80403ac20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff810006fd0 tx=0x7ff810006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:23.121 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.124+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff80c00b960 con 0x7ff81c0ff630 2024-03-26T21:58:23.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.394+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff81c062430 con 0x7ff81c0ff630 2024-03-26T21:58:23.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.395+0000 7ff8197fa700 1 -- 172.21.15.196:0/3850099361 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff80c027030 con 0x7ff81c0ff630 2024-03-26T21:58:23.392 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:23.393 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:23.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff804038760 msgr2=0x7ff80403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff804038760 0x7ff80403ac20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff810006fd0 tx=0x7ff810006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 msgr2=0x7ff81c0fc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0fc9f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff80c006010 tx=0x7ff80c009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 shutdown_connections 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff804038760 0x7ff80403ac20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 --2- 172.21.15.196:0/3850099361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff81c0ff630 0x7ff81c0fc9f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 >> 172.21.15.196:0/3850099361 conn(0x7ff81c0fabd0 msgr2=0x7ff81c068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 shutdown_connections 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:23.397+0000 7ff82242c700 1 -- 172.21.15.196:0/3850099361 wait complete. 2024-03-26T21:58:23.395 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:24.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:23 smithi194 ceph-mon[29693]: pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:24.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:23 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3850099361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:24.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:24.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:25.161 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:25.206 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:25.412 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:25 smithi194 ceph-mon[29693]: pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:26.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 -- 172.21.15.196:0/637253522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 msgr2=0x7f31dc100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:26.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 --2- 172.21.15.196:0/637253522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc100b30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009910 tx=0x7f31d8009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:26.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 -- 172.21.15.196:0/637253522 shutdown_connections 2024-03-26T21:58:26.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 --2- 172.21.15.196:0/637253522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc100b30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 -- 172.21.15.196:0/637253522 >> 172.21.15.196:0/637253522 conn(0x7f31dc0fbcd0 msgr2=0x7f31dc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 -- 172.21.15.196:0/637253522 shutdown_connections 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 -- 172.21.15.196:0/637253522 wait complete. 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 Processor -- start 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.717+0000 7f31e36f6700 1 -- start start 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e36f6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e36f6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc196120 con 0x7f31dc100710 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e1492700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e1492700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34372/0 (socket says 172.21.15.196:34372) 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e1492700 1 -- 172.21.15.196:0/2217712995 learned_addr learned my addr 172.21.15.196:0/2217712995 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e1492700 1 -- 172.21.15.196:0/2217712995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d80095f0 con 0x7f31dc100710 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31e1492700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc195be0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f31d8006010 tx=0x7f31d8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:26.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.718+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31d801d070 con 0x7f31dc100710 2024-03-26T21:58:26.718 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.719+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc196320 con 0x7f31dc100710 2024-03-26T21:58:26.718 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.719+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31d8005dc0 con 0x7f31dc100710 2024-03-26T21:58:26.718 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.719+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31d8022e90 con 0x7f31dc100710 2024-03-26T21:58:26.718 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.719+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc1967c0 con 0x7f31dc100710 2024-03-26T21:58:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.719+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f31d8022470 con 0x7f31dc100710 2024-03-26T21:58:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.719+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc18f540 con 0x7f31dc100710 2024-03-26T21:58:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.720+0000 7f31d27fc700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31c8038760 0x7f31c803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.720+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f31d804c8b0 con 0x7f31dc100710 2024-03-26T21:58:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.720+0000 7f31e0c91700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31c8038760 0x7f31c803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.720+0000 7f31e0c91700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31c8038760 0x7f31c803ac20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f31cc006fd0 tx=0x7f31cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:26.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.725+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31d800b960 con 0x7f31dc100710 2024-03-26T21:58:26.990 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.993+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31dc0623e0 con 0x7f31dc100710 2024-03-26T21:58:26.991 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.994+0000 7f31d27fc700 1 -- 172.21.15.196:0/2217712995 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f31dc0623e0 con 0x7f31dc100710 2024-03-26T21:58:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:26.991 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31c8038760 msgr2=0x7f31c803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31c8038760 0x7f31c803ac20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f31cc006fd0 tx=0x7f31cc006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 msgr2=0x7f31dc195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc195be0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f31d8006010 tx=0x7f31d8009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 shutdown_connections 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31c8038760 0x7f31c803ac20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 --2- 172.21.15.196:0/2217712995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31dc100710 0x7f31dc195be0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 >> 172.21.15.196:0/2217712995 conn(0x7f31dc0fbcd0 msgr2=0x7f31dc0fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 shutdown_connections 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:26.996+0000 7f31e36f6700 1 -- 172.21.15.196:0/2217712995 wait complete. 2024-03-26T21:58:26.994 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:28.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:27 smithi194 ceph-mon[29693]: pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:28.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:27 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2217712995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:28.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:28.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:28.747 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:30.153 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:29 smithi194 ceph-mon[29693]: pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:30.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 -- 172.21.15.196:0/1033387891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 msgr2=0x7fd6540ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 --2- 172.21.15.196:0/1033387891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd6540ffa40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd644009910 tx=0x7fd644009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 -- 172.21.15.196:0/1033387891 shutdown_connections 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 --2- 172.21.15.196:0/1033387891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd6540ffa40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 -- 172.21.15.196:0/1033387891 >> 172.21.15.196:0/1033387891 conn(0x7fd6540fab80 msgr2=0x7fd6540fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 -- 172.21.15.196:0/1033387891 shutdown_connections 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.325+0000 7fd658d2e700 1 -- 172.21.15.196:0/1033387891 wait complete. 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd658d2e700 1 Processor -- start 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd658d2e700 1 -- start start 2024-03-26T21:58:30.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd658d2e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd654197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd658d2e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654198350 con 0x7fd6540ff620 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd65259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd654197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd65259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd654197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34396/0 (socket says 172.21.15.196:34396) 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.326+0000 7fd65259c700 1 -- 172.21.15.196:0/3384209937 learned_addr learned my addr 172.21.15.196:0/3384209937 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd65259c700 1 -- 172.21.15.196:0/3384209937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6440095f0 con 0x7fd6540ff620 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd65259c700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd654197e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd644006010 tx=0x7fd644009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd64401d070 con 0x7fd6540ff620 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd654198550 con 0x7fd6540ff620 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd644005dc0 con 0x7fd6540ff620 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd644022e90 con 0x7fd6540ff620 2024-03-26T21:58:30.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.327+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6541989f0 con 0x7fd6540ff620 2024-03-26T21:58:30.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.328+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fd64400f570 con 0x7fd6540ff620 2024-03-26T21:58:30.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.328+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6540623e0 con 0x7fd6540ff620 2024-03-26T21:58:30.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.328+0000 7fd6437fe700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd63c0387b0 0x7fd63c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:30.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.328+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd64404cc80 con 0x7fd6540ff620 2024-03-26T21:58:30.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.328+0000 7fd651d9b700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd63c0387b0 0x7fd63c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:30.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.329+0000 7fd651d9b700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd63c0387b0 0x7fd63c03ac70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd648006fd0 tx=0x7fd648006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:30.331 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.334+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd64402a950 con 0x7fd6540ff620 2024-03-26T21:58:30.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.604+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd654105650 con 0x7fd6540ff620 2024-03-26T21:58:30.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.605+0000 7fd6437fe700 1 -- 172.21.15.196:0/3384209937 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd644027020 con 0x7fd6540ff620 2024-03-26T21:58:30.603 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:30.603 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd63c0387b0 msgr2=0x7fd63c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd63c0387b0 0x7fd63c03ac70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd648006fd0 tx=0x7fd648006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 msgr2=0x7fd654197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd654197e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd644006010 tx=0x7fd644009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 shutdown_connections 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd63c0387b0 0x7fd63c03ac70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 --2- 172.21.15.196:0/3384209937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6540ff620 0x7fd654197e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 >> 172.21.15.196:0/3384209937 conn(0x7fd6540fab80 msgr2=0x7fd6540683e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 shutdown_connections 2024-03-26T21:58:30.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:30.607+0000 7fd658d2e700 1 -- 172.21.15.196:0/3384209937 wait complete. 2024-03-26T21:58:30.606 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:31.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:30 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3384209937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:32.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:31 smithi194 ceph-mon[29693]: pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:32.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:32.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:32.379 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:32.425 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:33.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 -- 172.21.15.196:0/3656733348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 msgr2=0x7ff734101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:33.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 --2- 172.21.15.196:0/3656733348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff734101060 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff728009910 tx=0x7ff728009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:33.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 -- 172.21.15.196:0/3656733348 shutdown_connections 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 --2- 172.21.15.196:0/3656733348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff734101060 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 -- 172.21.15.196:0/3656733348 >> 172.21.15.196:0/3656733348 conn(0x7ff7340fa380 msgr2=0x7ff7340fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 -- 172.21.15.196:0/3656733348 shutdown_connections 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.855+0000 7ff73897e700 1 -- 172.21.15.196:0/3656733348 wait complete. 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73897e700 1 Processor -- start 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73897e700 1 -- start start 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73897e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff7340ff210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73897e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340ff750 con 0x7ff734100c40 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff7340ff210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff7340ff210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44028/0 (socket says 172.21.15.196:44028) 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73259c700 1 -- 172.21.15.196:0/1185453896 learned_addr learned my addr 172.21.15.196:0/1185453896 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73259c700 1 -- 172.21.15.196:0/1185453896 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7280095f0 con 0x7ff734100c40 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.856+0000 7ff73259c700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff7340ff210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff728006010 tx=0x7ff728009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.857+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff72801d070 con 0x7ff734100c40 2024-03-26T21:58:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.857+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff734198940 con 0x7ff734100c40 2024-03-26T21:58:33.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.857+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff728005dc0 con 0x7ff734100c40 2024-03-26T21:58:33.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.857+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff728022e90 con 0x7ff734100c40 2024-03-26T21:58:33.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.857+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734198d80 con 0x7ff734100c40 2024-03-26T21:58:33.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.858+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73418e4a0 con 0x7ff734100c40 2024-03-26T21:58:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.858+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff728022470 con 0x7ff734100c40 2024-03-26T21:58:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.858+0000 7ff7237fe700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff71c038760 0x7ff71c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.858+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff72804c7b0 con 0x7ff734100c40 2024-03-26T21:58:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.858+0000 7ff731d9b700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff71c038760 0x7ff71c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.858+0000 7ff731d9b700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff71c038760 0x7ff71c03ac20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff724006fd0 tx=0x7ff724006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:33.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:33.863+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff72800baf0 con 0x7ff734100c40 2024-03-26T21:58:34.129 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.132+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff734101770 con 0x7ff734100c40 2024-03-26T21:58:34.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.132+0000 7ff7237fe700 1 -- 172.21.15.196:0/1185453896 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff734101770 con 0x7ff734100c40 2024-03-26T21:58:34.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:34.130 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:34.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff71c038760 msgr2=0x7ff71c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:34.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff71c038760 0x7ff71c03ac20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff724006fd0 tx=0x7ff724006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 msgr2=0x7ff7340ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff7340ff210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff728006010 tx=0x7ff728009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 shutdown_connections 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff71c038760 0x7ff71c03ac20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 --2- 172.21.15.196:0/1185453896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff734100c40 0x7ff7340ff210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 >> 172.21.15.196:0/1185453896 conn(0x7ff7340fa380 msgr2=0x7ff7340fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 shutdown_connections 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:34.135+0000 7ff73897e700 1 -- 172.21.15.196:0/1185453896 wait complete. 2024-03-26T21:58:34.133 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:34.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:33 smithi194 ceph-mon[29693]: pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:35.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:35 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1185453896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:35.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:35 smithi194 ceph-mon[29693]: pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:35.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:35.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:35.930 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:35.977 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 -- 172.21.15.196:0/742039070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 msgr2=0x7ff130100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 --2- 172.21.15.196:0/742039070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130100b30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff120009910 tx=0x7ff120009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 -- 172.21.15.196:0/742039070 shutdown_connections 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 --2- 172.21.15.196:0/742039070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130100b30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 -- 172.21.15.196:0/742039070 >> 172.21.15.196:0/742039070 conn(0x7ff1300fbcd0 msgr2=0x7ff1300fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 -- 172.21.15.196:0/742039070 shutdown_connections 2024-03-26T21:58:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 -- 172.21.15.196:0/742039070 wait complete. 2024-03-26T21:58:37.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 Processor -- start 2024-03-26T21:58:37.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.461+0000 7ff135f95700 1 -- start start 2024-03-26T21:58:37.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff135f95700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:37.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff135f95700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130196120 con 0x7ff130100710 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44038/0 (socket says 172.21.15.196:44038) 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff12f7fe700 1 -- 172.21.15.196:0/2222690431 learned_addr learned my addr 172.21.15.196:0/2222690431 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff12f7fe700 1 -- 172.21.15.196:0/2222690431 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1200095f0 con 0x7ff130100710 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff12f7fe700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130195be0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff120006010 tx=0x7ff120009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.462+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12001d070 con 0x7ff130100710 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.463+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff130196320 con 0x7ff130100710 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.463+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff120005dc0 con 0x7ff130100710 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.463+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff120022e90 con 0x7ff130100710 2024-03-26T21:58:37.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.463+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1301967c0 con 0x7ff130100710 2024-03-26T21:58:37.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.463+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff12000f510 con 0x7ff130100710 2024-03-26T21:58:37.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.464+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13018f540 con 0x7ff130100710 2024-03-26T21:58:37.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.464+0000 7ff12cff9700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff118038760 0x7ff11803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:37.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.464+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff12004cb90 con 0x7ff130100710 2024-03-26T21:58:37.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.464+0000 7ff12effd700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff118038760 0x7ff11803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:37.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.464+0000 7ff12effd700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff118038760 0x7ff11803ac20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff124006fd0 tx=0x7ff124006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:37.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.469+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff12002a950 con 0x7ff130100710 2024-03-26T21:58:37.735 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.737+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1300623e0 con 0x7ff130100710 2024-03-26T21:58:37.735 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.738+0000 7ff12cff9700 1 -- 172.21.15.196:0/2222690431 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff120027020 con 0x7ff130100710 2024-03-26T21:58:37.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:37.735 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff118038760 msgr2=0x7ff11803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff118038760 0x7ff11803ac20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff124006fd0 tx=0x7ff124006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 msgr2=0x7ff130195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130195be0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff120006010 tx=0x7ff120009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 shutdown_connections 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff118038760 0x7ff11803ac20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 --2- 172.21.15.196:0/2222690431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff130100710 0x7ff130195be0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 >> 172.21.15.196:0/2222690431 conn(0x7ff1300fbcd0 msgr2=0x7ff1300fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 shutdown_connections 2024-03-26T21:58:37.738 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:37.740+0000 7ff135f95700 1 -- 172.21.15.196:0/2222690431 wait complete. 2024-03-26T21:58:37.739 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:38.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:37 smithi194 ceph-mon[29693]: pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:39.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:38 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2222690431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:39.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:39.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:39.538 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:39.584 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:40.064 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:39 smithi194 ceph-mon[29693]: pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:41.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 -- 172.21.15.196:0/2234806637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 msgr2=0x7f397c100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:41.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 --2- 172.21.15.196:0/2234806637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c100b70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3974009910 tx=0x7f3974009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:41.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 -- 172.21.15.196:0/2234806637 shutdown_connections 2024-03-26T21:58:41.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 --2- 172.21.15.196:0/2234806637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c100b70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 -- 172.21.15.196:0/2234806637 >> 172.21.15.196:0/2234806637 conn(0x7f397c0fbcf0 msgr2=0x7f397c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 -- 172.21.15.196:0/2234806637 shutdown_connections 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 -- 172.21.15.196:0/2234806637 wait complete. 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 Processor -- start 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.074+0000 7f3981515700 1 -- start start 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f3981515700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f3981515700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c196100 con 0x7f397c100750 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f397affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f397affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44058/0 (socket says 172.21.15.196:44058) 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f397affd700 1 -- 172.21.15.196:0/1561580750 learned_addr learned my addr 172.21.15.196:0/1561580750 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f397affd700 1 -- 172.21.15.196:0/1561580750 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740095f0 con 0x7f397c100750 2024-03-26T21:58:41.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f397affd700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c195bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3974006010 tx=0x7f3974009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f397401d070 con 0x7f397c100750 2024-03-26T21:58:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.075+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3974005dc0 con 0x7f397c100750 2024-03-26T21:58:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.076+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c196300 con 0x7f397c100750 2024-03-26T21:58:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.076+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3974022e90 con 0x7f397c100750 2024-03-26T21:58:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.076+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c1967a0 con 0x7f397c100750 2024-03-26T21:58:41.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.077+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f397400b960 con 0x7f397c100750 2024-03-26T21:58:41.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.077+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c18f580 con 0x7f397c100750 2024-03-26T21:58:41.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.077+0000 7f3963fff700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3964038760 0x7f396403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:41.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.077+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f397404c800 con 0x7f397c100750 2024-03-26T21:58:41.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.077+0000 7f397a7fc700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3964038760 0x7f396403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:41.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.077+0000 7f397a7fc700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3964038760 0x7f396403ac20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3970006fd0 tx=0x7f3970006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:41.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.082+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3974022750 con 0x7f397c100750 2024-03-26T21:58:41.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.351+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f397c0623e0 con 0x7f397c100750 2024-03-26T21:58:41.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.351+0000 7f3963fff700 1 -- 172.21.15.196:0/1561580750 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f397c0623e0 con 0x7f397c100750 2024-03-26T21:58:41.349 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:41.349 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:41.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3964038760 msgr2=0x7f396403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:41.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3964038760 0x7f396403ac20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3970006fd0 tx=0x7f3970006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:41.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 msgr2=0x7f397c195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:41.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c195bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3974006010 tx=0x7f3974009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 shutdown_connections 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3964038760 0x7f396403ac20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 --2- 172.21.15.196:0/1561580750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f397c100750 0x7f397c195bc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.353+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 >> 172.21.15.196:0/1561580750 conn(0x7f397c0fbcf0 msgr2=0x7f397c0fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.354+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 shutdown_connections 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:41.354+0000 7f3981515700 1 -- 172.21.15.196:0/1561580750 wait complete. 2024-03-26T21:58:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:42.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:41 smithi194 ceph-mon[29693]: pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:42.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:41 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1561580750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:43.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:43.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:43.207 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:43.253 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:44.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:43 smithi194 ceph-mon[29693]: pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 -- 172.21.15.196:0/3829216969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 msgr2=0x7ff47c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 --2- 172.21.15.196:0/3829216969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c0ffa30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009910 tx=0x7ff46c009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 -- 172.21.15.196:0/3829216969 shutdown_connections 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 --2- 172.21.15.196:0/3829216969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c0ffa30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 -- 172.21.15.196:0/3829216969 >> 172.21.15.196:0/3829216969 conn(0x7ff47c0fabb0 msgr2=0x7ff47c0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 -- 172.21.15.196:0/3829216969 shutdown_connections 2024-03-26T21:58:44.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.680+0000 7ff48198a700 1 -- 172.21.15.196:0/3829216969 wait complete. 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff48198a700 1 Processor -- start 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff48198a700 1 -- start start 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff48198a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c069160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff48198a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c0696a0 con 0x7ff47c0ff610 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff47affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c069160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff47affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c069160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:47272/0 (socket says 172.21.15.196:47272) 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.681+0000 7ff47affd700 1 -- 172.21.15.196:0/2849058637 learned_addr learned my addr 172.21.15.196:0/2849058637 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff47affd700 1 -- 172.21.15.196:0/2849058637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0095f0 con 0x7ff47c0ff610 2024-03-26T21:58:44.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff47affd700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c069160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff46c006010 tx=0x7ff46c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:44.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46c01d070 con 0x7ff47c0ff610 2024-03-26T21:58:44.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff46c005dc0 con 0x7ff47c0ff610 2024-03-26T21:58:44.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c198900 con 0x7ff47c0ff610 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46c022e90 con 0x7ff47c0ff610 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.682+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c198d40 con 0x7ff47c0ff610 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.683+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47c18e440 con 0x7ff47c0ff610 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.683+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff46c00b960 con 0x7ff47c0ff610 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.683+0000 7ff480988700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff464038760 0x7ff46403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.683+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff46c04c800 con 0x7ff47c0ff610 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.683+0000 7ff47a7fc700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff464038760 0x7ff46403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.684+0000 7ff47a7fc700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff464038760 0x7ff46403ac20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff470006fd0 tx=0x7ff470006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:44.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.688+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff46c00f4f0 con 0x7ff47c0ff610 2024-03-26T21:58:44.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.957+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff47c0623e0 con 0x7ff47c0ff610 2024-03-26T21:58:44.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.957+0000 7ff480988700 1 -- 172.21.15.196:0/2849058637 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff46c027030 con 0x7ff47c0ff610 2024-03-26T21:58:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:44.955 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:44.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.959+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff464038760 msgr2=0x7ff46403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:44.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.959+0000 7ff48198a700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff464038760 0x7ff46403ac20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff470006fd0 tx=0x7ff470006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.959+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 msgr2=0x7ff47c069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c069160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff46c006010 tx=0x7ff46c009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 shutdown_connections 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff464038760 0x7ff46403ac20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 --2- 172.21.15.196:0/2849058637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff47c0ff610 0x7ff47c069160 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 >> 172.21.15.196:0/2849058637 conn(0x7ff47c0fabb0 msgr2=0x7ff47c0fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 shutdown_connections 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:44.960+0000 7ff48198a700 1 -- 172.21.15.196:0/2849058637 wait complete. 2024-03-26T21:58:44.958 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:45.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:45 smithi194 ceph-mon[29693]: pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:45.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:45 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2849058637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:46.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:46.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:46.722 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:46.769 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:48.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:47 smithi194 ceph-mon[29693]: pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.218+0000 7f34950d7700 1 -- 172.21.15.196:0/1582660712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 msgr2=0x7f3490100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.218+0000 7f34950d7700 1 --2- 172.21.15.196:0/1582660712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490100b40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3480009910 tx=0x7f3480009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.218+0000 7f34950d7700 1 -- 172.21.15.196:0/1582660712 shutdown_connections 2024-03-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.218+0000 7f34950d7700 1 --2- 172.21.15.196:0/1582660712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490100b40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.218+0000 7f34950d7700 1 -- 172.21.15.196:0/1582660712 >> 172.21.15.196:0/1582660712 conn(0x7f34900fbcc0 msgr2=0x7f34900fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.218+0000 7f34950d7700 1 -- 172.21.15.196:0/1582660712 shutdown_connections 2024-03-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.219+0000 7f34950d7700 1 -- 172.21.15.196:0/1582660712 wait complete. 2024-03-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.219+0000 7f34950d7700 1 Processor -- start 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.219+0000 7f34950d7700 1 -- start start 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.219+0000 7f34950d7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.220+0000 7f34950d7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3490074f60 con 0x7f3490100720 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.220+0000 7f348ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.220+0000 7f348ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:47294/0 (socket says 172.21.15.196:47294) 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.220+0000 7f348ed9d700 1 -- 172.21.15.196:0/1465762305 learned_addr learned my addr 172.21.15.196:0/1465762305 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.220+0000 7f348ed9d700 1 -- 172.21.15.196:0/1465762305 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34800095f0 con 0x7f3490100720 2024-03-26T21:58:48.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.220+0000 7f348ed9d700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490074a20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3480006010 tx=0x7f3480009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:48.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.221+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f348001d070 con 0x7f3490100720 2024-03-26T21:58:48.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.221+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34900731a0 con 0x7f3490100720 2024-03-26T21:58:48.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.221+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3480005dc0 con 0x7f3490100720 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.221+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3480022e90 con 0x7f3490100720 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.221+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3490073640 con 0x7f3490100720 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.222+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f348000f4f0 con 0x7f3490100720 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.222+0000 7f3477fff700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3478038760 0x7f347803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.222+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3490106e80 con 0x7f3490100720 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.222+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f348004c8e0 con 0x7f3490100720 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.222+0000 7f348e59c700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3478038760 0x7f347803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:48.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.222+0000 7f348e59c700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3478038760 0x7f347803ac20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3484006fd0 tx=0x7f3484006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:48.225 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.227+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3480027020 con 0x7f3490100720 2024-03-26T21:58:48.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.494+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34900623e0 con 0x7f3490100720 2024-03-26T21:58:48.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.495+0000 7f3477fff700 1 -- 172.21.15.196:0/1465762305 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f348002a950 con 0x7f3490100720 2024-03-26T21:58:48.493 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:48.493 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:48.494 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3478038760 msgr2=0x7f347803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3478038760 0x7f347803ac20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3484006fd0 tx=0x7f3484006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 msgr2=0x7f3490074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490074a20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3480006010 tx=0x7f3480009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 shutdown_connections 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3478038760 0x7f347803ac20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 --2- 172.21.15.196:0/1465762305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3490100720 0x7f3490074a20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 >> 172.21.15.196:0/1465762305 conn(0x7f34900fbcc0 msgr2=0x7f34900fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 shutdown_connections 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:48.497+0000 7f34950d7700 1 -- 172.21.15.196:0/1465762305 wait complete. 2024-03-26T21:58:48.495 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:49.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:48 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1465762305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:50.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:50.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:50.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:49 smithi194 ceph-mon[29693]: pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:50.248 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:50.294 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 -- 172.21.15.196:0/1290528068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 msgr2=0x7f644c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 --2- 172.21.15.196:0/1290528068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c100b40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f643c00b0a0 tx=0x7f643c00b3b0 comp rx=0 tx=0).stop 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 -- 172.21.15.196:0/1290528068 shutdown_connections 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 --2- 172.21.15.196:0/1290528068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c100b40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 -- 172.21.15.196:0/1290528068 >> 172.21.15.196:0/1290528068 conn(0x7f644c0fbcc0 msgr2=0x7f644c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 -- 172.21.15.196:0/1290528068 shutdown_connections 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 -- 172.21.15.196:0/1290528068 wait complete. 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.912+0000 7f6451d7d700 1 Processor -- start 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f6451d7d700 1 -- start start 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f6451d7d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f6451d7d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c074f60 con 0x7f644c100720 2024-03-26T21:58:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f644b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:51.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f644b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:47308/0 (socket says 172.21.15.196:47308) 2024-03-26T21:58:51.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f644b7fe700 1 -- 172.21.15.196:0/2912111916 learned_addr learned my addr 172.21.15.196:0/2912111916 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:51.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f644b7fe700 1 -- 172.21.15.196:0/2912111916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c009d00 con 0x7f644c100720 2024-03-26T21:58:51.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.913+0000 7f644b7fe700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c074a20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f643c0062a0 tx=0x7f643c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:51.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.914+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f643c00e040 con 0x7f644c100720 2024-03-26T21:58:51.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.914+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c0731a0 con 0x7f644c100720 2024-03-26T21:58:51.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.914+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f643c003d70 con 0x7f644c100720 2024-03-26T21:58:51.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.914+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f643c01c890 con 0x7f644c100720 2024-03-26T21:58:51.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.914+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c073640 con 0x7f644c100720 2024-03-26T21:58:51.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.915+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f643c01cb50 con 0x7f644c100720 2024-03-26T21:58:51.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.915+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c106e80 con 0x7f644c100720 2024-03-26T21:58:51.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.915+0000 7f6448ff9700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64340387c0 0x7f643403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:51.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.915+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f643c04ca20 con 0x7f644c100720 2024-03-26T21:58:51.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.915+0000 7f644affd700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64340387c0 0x7f643403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:51.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.915+0000 7f644affd700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64340387c0 0x7f643403ac80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6440006fd0 tx=0x7f6440006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:51.918 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:51.920+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f643c01a3b0 con 0x7f644c100720 2024-03-26T21:58:52.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:51 smithi194 ceph-mon[29693]: pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:52.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.190+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f644c0623e0 con 0x7f644c100720 2024-03-26T21:58:52.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.191+0000 7f6448ff9700 1 -- 172.21.15.196:0/2912111916 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f643c01a3b0 con 0x7f644c100720 2024-03-26T21:58:52.189 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:52.189 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64340387c0 msgr2=0x7f643403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64340387c0 0x7f643403ac80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6440006fd0 tx=0x7f6440006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 msgr2=0x7f644c074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c074a20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f643c0062a0 tx=0x7f643c00bd40 comp rx=0 tx=0).stop 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 shutdown_connections 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f64340387c0 0x7f643403ac80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 --2- 172.21.15.196:0/2912111916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f644c100720 0x7f644c074a20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 >> 172.21.15.196:0/2912111916 conn(0x7f644c0fbcc0 msgr2=0x7f644c0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:52.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 shutdown_connections 2024-03-26T21:58:52.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:52.193+0000 7f6451d7d700 1 -- 172.21.15.196:0/2912111916 wait complete. 2024-03-26T21:58:52.192 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:53.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:52 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2912111916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:53.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:53.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:53.969 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:54.015 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:54.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:53 smithi194 ceph-mon[29693]: pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:55.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:55 smithi194 ceph-mon[29693]: pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:55.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 -- 172.21.15.196:0/1505856372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 msgr2=0x7f9818100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:55.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 --2- 172.21.15.196:0/1505856372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818100b60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9808009910 tx=0x7f9808009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:55.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 -- 172.21.15.196:0/1505856372 shutdown_connections 2024-03-26T21:58:55.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 --2- 172.21.15.196:0/1505856372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818100b60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 -- 172.21.15.196:0/1505856372 >> 172.21.15.196:0/1505856372 conn(0x7f98180fbd00 msgr2=0x7f98180fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 -- 172.21.15.196:0/1505856372 shutdown_connections 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 -- 172.21.15.196:0/1505856372 wait complete. 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 Processor -- start 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.514+0000 7f981f5e6700 1 -- start start 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981f5e6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981f5e6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9818074f60 con 0x7f9818100740 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981d382700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981d382700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56500/0 (socket says 172.21.15.196:56500) 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981d382700 1 -- 172.21.15.196:0/1740372252 learned_addr learned my addr 172.21.15.196:0/1740372252 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981d382700 1 -- 172.21.15.196:0/1740372252 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98080095f0 con 0x7f9818100740 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.515+0000 7f981d382700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818074a20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9808006010 tx=0x7f9808009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.516+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980801d070 con 0x7f9818100740 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.516+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9808005dc0 con 0x7f9818100740 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.516+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98180731a0 con 0x7f9818100740 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.516+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9808022e90 con 0x7f9818100740 2024-03-26T21:58:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.516+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9818073640 con 0x7f9818100740 2024-03-26T21:58:55.516 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.516+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9818106ea0 con 0x7f9818100740 2024-03-26T21:58:55.516 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.517+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9808022470 con 0x7f9818100740 2024-03-26T21:58:55.516 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.517+0000 7f980e7fc700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9804038760 0x7f980403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:55.516 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.517+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f980804c920 con 0x7f9818100740 2024-03-26T21:58:55.516 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.517+0000 7f981cb81700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9804038760 0x7f980403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:55.516 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.517+0000 7f981cb81700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9804038760 0x7f980403ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9814006fd0 tx=0x7f9814006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:55.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.522+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f980800b960 con 0x7f9818100740 2024-03-26T21:58:55.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.794+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98180623e0 con 0x7f9818100740 2024-03-26T21:58:55.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.794+0000 7f980e7fc700 1 -- 172.21.15.196:0/1740372252 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9808027020 con 0x7f9818100740 2024-03-26T21:58:55.792 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:55.792 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:55.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9804038760 msgr2=0x7f980403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:55.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9804038760 0x7f980403ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9814006fd0 tx=0x7f9814006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:55.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 msgr2=0x7f9818074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:55.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818074a20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9808006010 tx=0x7f9808009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 shutdown_connections 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9804038760 0x7f980403ac20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 --2- 172.21.15.196:0/1740372252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9818100740 0x7f9818074a20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 >> 172.21.15.196:0/1740372252 conn(0x7f98180fbd00 msgr2=0x7f98180fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 shutdown_connections 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:55.796+0000 7f981f5e6700 1 -- 172.21.15.196:0/1740372252 wait complete. 2024-03-26T21:58:55.795 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:58:56.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:56 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1740372252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:58:57.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:58:57.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:58:57.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:57 smithi194 ceph-mon[29693]: pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:58:57.569 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:57.614 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.037+0000 7fe73cf94700 1 -- 172.21.15.196:0/728775453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 msgr2=0x7fe7380ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.037+0000 7fe73cf94700 1 --2- 172.21.15.196:0/728775453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe7380ffa40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009910 tx=0x7fe72c009c20 comp rx=0 tx=0).stop 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 -- 172.21.15.196:0/728775453 shutdown_connections 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 --2- 172.21.15.196:0/728775453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe7380ffa40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 -- 172.21.15.196:0/728775453 >> 172.21.15.196:0/728775453 conn(0x7fe7380fab80 msgr2=0x7fe7380fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 -- 172.21.15.196:0/728775453 shutdown_connections 2024-03-26T21:58:59.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 -- 172.21.15.196:0/728775453 wait complete. 2024-03-26T21:58:59.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 Processor -- start 2024-03-26T21:58:59.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 -- start start 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe738193a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.038+0000 7fe73cf94700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe738193f50 con 0x7fe7380ff620 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.039+0000 7fe73659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe738193a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.039+0000 7fe73659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe738193a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56522/0 (socket says 172.21.15.196:56522) 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.039+0000 7fe73659c700 1 -- 172.21.15.196:0/3040499852 learned_addr learned my addr 172.21.15.196:0/3040499852 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.039+0000 7fe73659c700 1 -- 172.21.15.196:0/3040499852 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c0095f0 con 0x7fe7380ff620 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.039+0000 7fe73659c700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe738193a10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe72c006010 tx=0x7fe72c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.039+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe72c01d070 con 0x7fe7380ff620 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.040+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe738194150 con 0x7fe7380ff620 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.040+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe72c005dc0 con 0x7fe7380ff620 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.040+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe72c022e90 con 0x7fe7380ff620 2024-03-26T21:58:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.040+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7381945f0 con 0x7fe7380ff620 2024-03-26T21:58:59.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.040+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe72c00f4b0 con 0x7fe7380ff620 2024-03-26T21:58:59.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.041+0000 7fe7277fe700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe72003aa00 0x7fe72003cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:58:59.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.041+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe72c04c710 con 0x7fe7380ff620 2024-03-26T21:58:59.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.041+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73818d340 con 0x7fe7380ff620 2024-03-26T21:58:59.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.041+0000 7fe735d9b700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe72003aa00 0x7fe72003cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:58:59.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.041+0000 7fe735d9b700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe72003aa00 0x7fe72003cec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe728006fd0 tx=0x7fe728006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:58:59.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.047+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe72c027070 con 0x7fe7380ff620 2024-03-26T21:58:59.313 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.315+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7380623e0 con 0x7fe7380ff620 2024-03-26T21:58:59.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.316+0000 7fe7277fe700 1 -- 172.21.15.196:0/3040499852 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe72c034070 con 0x7fe7380ff620 2024-03-26T21:58:59.314 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:58:59.314 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe72003aa00 msgr2=0x7fe72003cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe72003aa00 0x7fe72003cec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe728006fd0 tx=0x7fe728006e40 comp rx=0 tx=0).stop 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 msgr2=0x7fe738193a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe738193a10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe72c006010 tx=0x7fe72c009f80 comp rx=0 tx=0).stop 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 shutdown_connections 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe72003aa00 0x7fe72003cec0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 --2- 172.21.15.196:0/3040499852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7380ff620 0x7fe738193a10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 >> 172.21.15.196:0/3040499852 conn(0x7fe7380fab80 msgr2=0x7fe7380fb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 shutdown_connections 2024-03-26T21:58:59.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:58:59.318+0000 7fe73cf94700 1 -- 172.21.15.196:0/3040499852 wait complete. 2024-03-26T21:58:59.317 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:00.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:59 smithi194 ceph-mon[29693]: pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:00.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:58:59 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3040499852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:00.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:00.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:01.118 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:01.163 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:02.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:01 smithi194 ceph-mon[29693]: pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 -- 172.21.15.196:0/4219290386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 msgr2=0x7f29041013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 --2- 172.21.15.196:0/4219290386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f29041013f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009910 tx=0x7f28f4009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 -- 172.21.15.196:0/4219290386 shutdown_connections 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 --2- 172.21.15.196:0/4219290386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f29041013f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 -- 172.21.15.196:0/4219290386 >> 172.21.15.196:0/4219290386 conn(0x7f29040fabe0 msgr2=0x7f29040fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 -- 172.21.15.196:0/4219290386 shutdown_connections 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.637+0000 7f2908aef700 1 -- 172.21.15.196:0/4219290386 wait complete. 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f2908aef700 1 Processor -- start 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f2908aef700 1 -- start start 2024-03-26T21:59:02.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f2908aef700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f290419c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f2908aef700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290419c770 con 0x7f29040fefc0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f290259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f290419c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f290259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f290419c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43816/0 (socket says 172.21.15.196:43816) 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f290259c700 1 -- 172.21.15.196:0/2703035219 learned_addr learned my addr 172.21.15.196:0/2703035219 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f290259c700 1 -- 172.21.15.196:0/2703035219 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40095f0 con 0x7f29040fefc0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.638+0000 7f290259c700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f290419c230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005b40 tx=0x7f28f4005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.639+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28f401d070 con 0x7f29040fefc0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.639+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28f4005dc0 con 0x7f29040fefc0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.639+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28f4022e00 con 0x7f29040fefc0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.639+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290419c970 con 0x7f29040fefc0 2024-03-26T21:59:02.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.639+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290419ce10 con 0x7f29040fefc0 2024-03-26T21:59:02.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.640+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f28f400b960 con 0x7f29040fefc0 2024-03-26T21:59:02.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.640+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29041962b0 con 0x7f29040fefc0 2024-03-26T21:59:02.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.640+0000 7f28f37fe700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28ec038760 0x7f28ec03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:02.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.640+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f28f404c920 con 0x7f29040fefc0 2024-03-26T21:59:02.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.640+0000 7f2901d9b700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28ec038760 0x7f28ec03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:02.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.640+0000 7f2901d9b700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28ec038760 0x7f28ec03ac20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f28f8006fd0 tx=0x7f28f8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:02.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.645+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f28f4022690 con 0x7f29040fefc0 2024-03-26T21:59:02.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.911+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29040623e0 con 0x7f29040fefc0 2024-03-26T21:59:02.910 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.912+0000 7f28f37fe700 1 -- 172.21.15.196:0/2703035219 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f29040623e0 con 0x7f29040fefc0 2024-03-26T21:59:02.910 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:02.910 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:02.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28ec038760 msgr2=0x7f28ec03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:02.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28ec038760 0x7f28ec03ac20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f28f8006fd0 tx=0x7f28f8006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:02.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 msgr2=0x7f290419c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:02.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f290419c230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005b40 tx=0x7f28f4005000 comp rx=0 tx=0).stop 2024-03-26T21:59:02.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 shutdown_connections 2024-03-26T21:59:02.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28ec038760 0x7f28ec03ac20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:02.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 --2- 172.21.15.196:0/2703035219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29040fefc0 0x7f290419c230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:02.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 >> 172.21.15.196:0/2703035219 conn(0x7f29040fabe0 msgr2=0x7f29040fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:02.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 shutdown_connections 2024-03-26T21:59:02.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:02.914+0000 7f2908aef700 1 -- 172.21.15.196:0/2703035219 wait complete. 2024-03-26T21:59:02.913 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:04.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:03 smithi194 ceph-mon[29693]: pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:04.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:03 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2703035219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:04.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:04.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:04.736 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:04.782 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:05.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:05 smithi194 ceph-mon[29693]: pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 -- 172.21.15.196:0/1025123800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 msgr2=0x7ff5440ff880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 --2- 172.21.15.196:0/1025123800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff5440ff880 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff534009910 tx=0x7ff534009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 -- 172.21.15.196:0/1025123800 shutdown_connections 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 --2- 172.21.15.196:0/1025123800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff5440ff880 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 -- 172.21.15.196:0/1025123800 >> 172.21.15.196:0/1025123800 conn(0x7ff5440faa00 msgr2=0x7ff5440fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 -- 172.21.15.196:0/1025123800 shutdown_connections 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.299+0000 7ff5489fe700 1 -- 172.21.15.196:0/1025123800 wait complete. 2024-03-26T21:59:06.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff5489fe700 1 Processor -- start 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff5489fe700 1 -- start start 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff5489fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff544195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff5489fe700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff544195fa0 con 0x7ff5440ff460 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff542d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff544195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff542d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff544195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43828/0 (socket says 172.21.15.196:43828) 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.300+0000 7ff542d9d700 1 -- 172.21.15.196:0/1883640081 learned_addr learned my addr 172.21.15.196:0/1883640081 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff542d9d700 1 -- 172.21.15.196:0/1883640081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5340095f0 con 0x7ff5440ff460 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff542d9d700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff544195a60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff534006010 tx=0x7ff534009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53401d070 con 0x7ff5440ff460 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5441961a0 con 0x7ff5440ff460 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff534005dc0 con 0x7ff5440ff460 2024-03-26T21:59:06.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff544196640 con 0x7ff5440ff460 2024-03-26T21:59:06.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.301+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff534022e50 con 0x7ff5440ff460 2024-03-26T21:59:06.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.302+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5440623e0 con 0x7ff5440ff460 2024-03-26T21:59:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.302+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff534022470 con 0x7ff5440ff460 2024-03-26T21:59:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.302+0000 7ff52bfff700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff52c038770 0x7ff52c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.302+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff53404c800 con 0x7ff5440ff460 2024-03-26T21:59:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.302+0000 7ff54259c700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff52c038770 0x7ff52c03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.303+0000 7ff54259c700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff52c038770 0x7ff52c03ac30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff538006fd0 tx=0x7ff538006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:06.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.307+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff53400b960 con 0x7ff5440ff460 2024-03-26T21:59:06.574 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.577+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff544105490 con 0x7ff5440ff460 2024-03-26T21:59:06.575 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.577+0000 7ff52bfff700 1 -- 172.21.15.196:0/1883640081 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff534027020 con 0x7ff5440ff460 2024-03-26T21:59:06.575 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:06.575 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:06.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff52c038770 msgr2=0x7ff52c03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:06.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff52c038770 0x7ff52c03ac30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff538006fd0 tx=0x7ff538006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:06.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 msgr2=0x7ff544195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:06.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff544195a60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff534006010 tx=0x7ff534009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 shutdown_connections 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff52c038770 0x7ff52c03ac30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 --2- 172.21.15.196:0/1883640081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5440ff460 0x7ff544195a60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 >> 172.21.15.196:0/1883640081 conn(0x7ff5440faa00 msgr2=0x7ff5440fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 shutdown_connections 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:06.579+0000 7ff5489fe700 1 -- 172.21.15.196:0/1883640081 wait complete. 2024-03-26T21:59:06.578 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:06.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:06 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1883640081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:07.913 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:07 smithi194 ceph-mon[29693]: pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:08.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:08.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:08.423 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:08.469 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 -- 172.21.15.196:0/3776693738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 msgr2=0x7f2d5c100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 --2- 172.21.15.196:0/3776693738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c100980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009910 tx=0x7f2d4c009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 -- 172.21.15.196:0/3776693738 shutdown_connections 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 --2- 172.21.15.196:0/3776693738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c100980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 -- 172.21.15.196:0/3776693738 >> 172.21.15.196:0/3776693738 conn(0x7f2d5c0fbb20 msgr2=0x7f2d5c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 -- 172.21.15.196:0/3776693738 shutdown_connections 2024-03-26T21:59:09.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 -- 172.21.15.196:0/3776693738 wait complete. 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.957+0000 7f2d6159f700 1 Processor -- start 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.958+0000 7f2d6159f700 1 -- start start 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.958+0000 7f2d6159f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.958+0000 7f2d6159f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c198140 con 0x7f2d5c100560 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.958+0000 7f2d5bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.958+0000 7f2d5bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43852/0 (socket says 172.21.15.196:43852) 2024-03-26T21:59:09.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.958+0000 7f2d5bfff700 1 -- 172.21.15.196:0/3190428868 learned_addr learned my addr 172.21.15.196:0/3190428868 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d5bfff700 1 -- 172.21.15.196:0/3190428868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c0095f0 con 0x7f2d5c100560 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d5bfff700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c197c00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c006010 tx=0x7f2d4c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4c01d070 con 0x7f2d5c100560 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c198340 con 0x7f2d5c100560 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d4c005dc0 con 0x7f2d5c100560 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4c022e90 con 0x7f2d5c100560 2024-03-26T21:59:09.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.959+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c1987e0 con 0x7f2d5c100560 2024-03-26T21:59:09.959 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.960+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2d4c00b960 con 0x7f2d5c100560 2024-03-26T21:59:09.959 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.960+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c191790 con 0x7f2d5c100560 2024-03-26T21:59:09.959 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.960+0000 7f2d597fa700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d440387c0 0x7f2d4403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:09.960 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.960+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2d4c04c840 con 0x7f2d5c100560 2024-03-26T21:59:09.960 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.960+0000 7f2d5b7fe700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d440387c0 0x7f2d4403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:09.960 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.961+0000 7f2d5b7fe700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d440387c0 0x7f2d4403ac80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2d50006fd0 tx=0x7f2d50006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:09.963 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:09.965+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2d4c022750 con 0x7f2d5c100560 2024-03-26T21:59:10.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:09 smithi194 ceph-mon[29693]: pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:10.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.232+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d5c0623e0 con 0x7f2d5c100560 2024-03-26T21:59:10.230 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.232+0000 7f2d597fa700 1 -- 172.21.15.196:0/3190428868 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2d5c0623e0 con 0x7f2d5c100560 2024-03-26T21:59:10.230 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:10.230 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:10.233 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d440387c0 msgr2=0x7f2d4403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:10.233 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d440387c0 0x7f2d4403ac80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2d50006fd0 tx=0x7f2d50006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:10.233 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 msgr2=0x7f2d5c197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c197c00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c006010 tx=0x7f2d4c009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 shutdown_connections 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d440387c0 0x7f2d4403ac80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 --2- 172.21.15.196:0/3190428868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2d5c100560 0x7f2d5c197c00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 >> 172.21.15.196:0/3190428868 conn(0x7f2d5c0fbb20 msgr2=0x7f2d5c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 shutdown_connections 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:10.235+0000 7f2d6159f700 1 -- 172.21.15.196:0/3190428868 wait complete. 2024-03-26T21:59:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:11.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:10 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3190428868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:11.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:11.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:11.999 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:12.045 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:12.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:11 smithi194 ceph-mon[29693]: pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:13.438 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 -- 172.21.15.196:0/650481195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 msgr2=0x7fe6240ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:13.438 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 --2- 172.21.15.196:0/650481195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6240ffa60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe614009910 tx=0x7fe614009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:13.438 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 -- 172.21.15.196:0/650481195 shutdown_connections 2024-03-26T21:59:13.438 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 --2- 172.21.15.196:0/650481195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6240ffa60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:13.438 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 -- 172.21.15.196:0/650481195 >> 172.21.15.196:0/650481195 conn(0x7fe6240fac00 msgr2=0x7fe6240fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:13.438 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 -- 172.21.15.196:0/650481195 shutdown_connections 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.438+0000 7fe62b32c700 1 -- 172.21.15.196:0/650481195 wait complete. 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe62b32c700 1 Processor -- start 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe62b32c700 1 -- start start 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe62b32c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6241939d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe62b32c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624193f10 con 0x7fe6240ff640 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe6290c8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6241939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe6290c8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6241939d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:38748/0 (socket says 172.21.15.196:38748) 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe6290c8700 1 -- 172.21.15.196:0/2901917693 learned_addr learned my addr 172.21.15.196:0/2901917693 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.439+0000 7fe6290c8700 1 -- 172.21.15.196:0/2901917693 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6140095f0 con 0x7fe6240ff640 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.440+0000 7fe6290c8700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6241939d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe614006010 tx=0x7fe614009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:13.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.440+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61401d070 con 0x7fe6240ff640 2024-03-26T21:59:13.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.440+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe624194110 con 0x7fe6240ff640 2024-03-26T21:59:13.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.440+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe614005dc0 con 0x7fe6240ff640 2024-03-26T21:59:13.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.440+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe614022e90 con 0x7fe6240ff640 2024-03-26T21:59:13.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.440+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6241945b0 con 0x7fe6240ff640 2024-03-26T21:59:13.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.441+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe61400f4b0 con 0x7fe6240ff640 2024-03-26T21:59:13.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.441+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62418d360 con 0x7fe6240ff640 2024-03-26T21:59:13.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.441+0000 7fe61a7fc700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe61003aa00 0x7fe61003cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:13.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.441+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe61404ca60 con 0x7fe6240ff640 2024-03-26T21:59:13.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.441+0000 7fe6288c7700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe61003aa00 0x7fe61003cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:13.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.442+0000 7fe6288c7700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe61003aa00 0x7fe61003cec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe620006fd0 tx=0x7fe620006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:13.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.446+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe614017440 con 0x7fe6240ff640 2024-03-26T21:59:13.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.715+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6240623e0 con 0x7fe6240ff640 2024-03-26T21:59:13.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.716+0000 7fe61a7fc700 1 -- 172.21.15.196:0/2901917693 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe6240623e0 con 0x7fe6240ff640 2024-03-26T21:59:13.714 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:13.714 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe61003aa00 msgr2=0x7fe61003cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe61003aa00 0x7fe61003cec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe620006fd0 tx=0x7fe620006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 msgr2=0x7fe6241939d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6241939d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe614006010 tx=0x7fe614009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 shutdown_connections 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe61003aa00 0x7fe61003cec0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 --2- 172.21.15.196:0/2901917693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6240ff640 0x7fe6241939d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:13.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 >> 172.21.15.196:0/2901917693 conn(0x7fe6240fac00 msgr2=0x7fe6240fb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:13.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 shutdown_connections 2024-03-26T21:59:13.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:13.718+0000 7fe62b32c700 1 -- 172.21.15.196:0/2901917693 wait complete. 2024-03-26T21:59:13.717 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:14.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:13 smithi194 ceph-mon[29693]: pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:15.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:14 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2901917693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:15.266 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:15.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:15.461 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:15.506 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:16.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:15 smithi194 ceph-mon[29693]: pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:17.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.014+0000 7f9bd1d71700 1 -- 172.21.15.196:0/1908618240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 msgr2=0x7f9bcc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:17.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.014+0000 7f9bd1d71700 1 --2- 172.21.15.196:0/1908618240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc100b40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009910 tx=0x7f9bbc009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:17.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 -- 172.21.15.196:0/1908618240 shutdown_connections 2024-03-26T21:59:17.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 --2- 172.21.15.196:0/1908618240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc100b40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:17.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 -- 172.21.15.196:0/1908618240 >> 172.21.15.196:0/1908618240 conn(0x7f9bcc0fbcc0 msgr2=0x7f9bcc0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:17.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 -- 172.21.15.196:0/1908618240 shutdown_connections 2024-03-26T21:59:17.013 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 -- 172.21.15.196:0/1908618240 wait complete. 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 Processor -- start 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.015+0000 7f9bd1d71700 1 -- start start 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.016+0000 7f9bd1d71700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.016+0000 7f9bd1d71700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc074f60 con 0x7f9bcc100720 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.016+0000 7f9bcb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.016+0000 7f9bcb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:38768/0 (socket says 172.21.15.196:38768) 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.016+0000 7f9bcb7fe700 1 -- 172.21.15.196:0/3969489140 learned_addr learned my addr 172.21.15.196:0/3969489140 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.016+0000 7f9bcb7fe700 1 -- 172.21.15.196:0/3969489140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0095f0 con 0x7f9bcc100720 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.017+0000 7f9bcb7fe700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc074a20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc006010 tx=0x7f9bbc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.017+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bbc01d070 con 0x7f9bcc100720 2024-03-26T21:59:17.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.017+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc0731a0 con 0x7f9bcc100720 2024-03-26T21:59:17.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.017+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bbc005dc0 con 0x7f9bcc100720 2024-03-26T21:59:17.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.017+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bbc022e90 con 0x7f9bcc100720 2024-03-26T21:59:17.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.017+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc073640 con 0x7f9bcc100720 2024-03-26T21:59:17.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.018+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc106e80 con 0x7f9bcc100720 2024-03-26T21:59:17.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.018+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9bbc022470 con 0x7f9bcc100720 2024-03-26T21:59:17.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.018+0000 7f9bc8ff9700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9bb4038760 0x7f9bb403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:17.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.018+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9bbc04c8b0 con 0x7f9bcc100720 2024-03-26T21:59:17.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.018+0000 7f9bcaffd700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9bb4038760 0x7f9bb403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:17.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.019+0000 7f9bcaffd700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9bb4038760 0x7f9bb403ac20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0006fd0 tx=0x7f9bc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:17.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.023+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9bbc00b960 con 0x7f9bcc100720 2024-03-26T21:59:17.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.290+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bcc0623e0 con 0x7f9bcc100720 2024-03-26T21:59:17.289 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.291+0000 7f9bc8ff9700 1 -- 172.21.15.196:0/3969489140 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9bbc027030 con 0x7f9bcc100720 2024-03-26T21:59:17.289 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:17.289 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:17.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9bb4038760 msgr2=0x7f9bb403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:17.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9bb4038760 0x7f9bb403ac20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0006fd0 tx=0x7f9bc0006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:17.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 msgr2=0x7f9bcc074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:17.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc074a20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc006010 tx=0x7f9bbc009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 shutdown_connections 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9bb4038760 0x7f9bb403ac20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 --2- 172.21.15.196:0/3969489140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bcc100720 0x7f9bcc074a20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 >> 172.21.15.196:0/3969489140 conn(0x7f9bcc0fbcc0 msgr2=0x7f9bcc0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 shutdown_connections 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:17.293+0000 7f9bd1d71700 1 -- 172.21.15.196:0/3969489140 wait complete. 2024-03-26T21:59:17.292 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:18.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:17 smithi194 ceph-mon[29693]: pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:18.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:17 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3969489140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:18.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:18.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:19.038 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:19.084 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:20.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:19 smithi194 ceph-mon[29693]: pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:20.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 -- 172.21.15.196:0/395417293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 msgr2=0x7f81840ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 --2- 172.21.15.196:0/395417293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f81840ffa40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b0a0 tx=0x7f817c00b3b0 comp rx=0 tx=0).stop 2024-03-26T21:59:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 -- 172.21.15.196:0/395417293 shutdown_connections 2024-03-26T21:59:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 --2- 172.21.15.196:0/395417293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f81840ffa40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 -- 172.21.15.196:0/395417293 >> 172.21.15.196:0/395417293 conn(0x7f81840fab80 msgr2=0x7f81840fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 -- 172.21.15.196:0/395417293 shutdown_connections 2024-03-26T21:59:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.498+0000 7f818ac8e700 1 -- 172.21.15.196:0/395417293 wait complete. 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.499+0000 7f818ac8e700 1 Processor -- start 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.499+0000 7f818ac8e700 1 -- start start 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.499+0000 7f818ac8e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f8184194ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.499+0000 7f818ac8e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8184195000 con 0x7f81840ff620 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8188a2a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f8184194ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8188a2a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f8184194ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:38782/0 (socket says 172.21.15.196:38782) 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8188a2a700 1 -- 172.21.15.196:0/2334825017 learned_addr learned my addr 172.21.15.196:0/2334825017 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8188a2a700 1 -- 172.21.15.196:0/2334825017 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c009d00 con 0x7f81840ff620 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8188a2a700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f8184194ac0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f817c0062a0 tx=0x7f817c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817c00e040 con 0x7f81840ff620 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8184195200 con 0x7f81840ff620 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f817c003d70 con 0x7f81840ff620 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.500+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817c019860 con 0x7f81840ff620 2024-03-26T21:59:20.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.501+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8184195700 con 0x7f81840ff620 2024-03-26T21:59:20.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.501+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81840623e0 con 0x7f81840ff620 2024-03-26T21:59:20.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.502+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f817c012640 con 0x7f81840ff620 2024-03-26T21:59:20.500 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.502+0000 7f8181ffb700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81700387c0 0x7f817003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:20.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.502+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f817c04e1c0 con 0x7f81840ff620 2024-03-26T21:59:20.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.502+0000 7f8183fff700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81700387c0 0x7f817003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:20.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.503+0000 7f8183fff700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81700387c0 0x7f817003ac80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8174006fd0 tx=0x7f8174006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:20.505 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.507+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f817c02a4e0 con 0x7f81840ff620 2024-03-26T21:59:20.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.776+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8184105460 con 0x7f81840ff620 2024-03-26T21:59:20.775 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.777+0000 7f8181ffb700 1 -- 172.21.15.196:0/2334825017 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8184105460 con 0x7f81840ff620 2024-03-26T21:59:20.775 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:20.775 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:20.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81700387c0 msgr2=0x7f817003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81700387c0 0x7f817003ac80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8174006fd0 tx=0x7f8174006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 msgr2=0x7f8184194ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f8184194ac0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f817c0062a0 tx=0x7f817c00bd40 comp rx=0 tx=0).stop 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 shutdown_connections 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f81700387c0 0x7f817003ac80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.779+0000 7f818ac8e700 1 --2- 172.21.15.196:0/2334825017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81840ff620 0x7f8184194ac0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.780+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 >> 172.21.15.196:0/2334825017 conn(0x7f81840fab80 msgr2=0x7f81840fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.780+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 shutdown_connections 2024-03-26T21:59:20.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:20.780+0000 7f818ac8e700 1 -- 172.21.15.196:0/2334825017 wait complete. 2024-03-26T21:59:20.779 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:22.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:21 smithi194 ceph-mon[29693]: pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:22.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:21 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2334825017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:22.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:22.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:22.576 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:22.621 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:24.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 -- 172.21.15.196:0/358732062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 msgr2=0x7fb8c0100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 --2- 172.21.15.196:0/358732062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0100b60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009910 tx=0x7fb8b0009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 -- 172.21.15.196:0/358732062 shutdown_connections 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 --2- 172.21.15.196:0/358732062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0100b60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 -- 172.21.15.196:0/358732062 >> 172.21.15.196:0/358732062 conn(0x7fb8c00fbd00 msgr2=0x7fb8c00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 -- 172.21.15.196:0/358732062 shutdown_connections 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 -- 172.21.15.196:0/358732062 wait complete. 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.012+0000 7fb8c55ce700 1 Processor -- start 2024-03-26T21:59:24.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8c55ce700 1 -- start start 2024-03-26T21:59:24.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8c55ce700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:24.012 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8c55ce700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0074f60 con 0x7fb8c0100740 2024-03-26T21:59:24.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:24.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36348/0 (socket says 172.21.15.196:36348) 2024-03-26T21:59:24.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8beffd700 1 -- 172.21.15.196:0/1670716499 learned_addr learned my addr 172.21.15.196:0/1670716499 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:24.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8beffd700 1 -- 172.21.15.196:0/1670716499 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b00095f0 con 0x7fb8c0100740 2024-03-26T21:59:24.014 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.013+0000 7fb8beffd700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0074a20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0006010 tx=0x7fb8b0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.014+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b001d070 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.014+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c00731a0 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.014+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8b0005dc0 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.014+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b0022e90 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.014+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c0073640 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.015+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c0106ea0 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.015+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb8b0022470 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.015+0000 7fb8a7fff700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb8a8038760 0x7fb8a803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.015+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb8b004c920 con 0x7fb8c0100740 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.015+0000 7fb8be7fc700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb8a8038760 0x7fb8a803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:24.015 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.016+0000 7fb8be7fc700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb8a8038760 0x7fb8a803ac20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4006fd0 tx=0x7fb8b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:24.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.020+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb8b000f4f0 con 0x7fb8c0100740 2024-03-26T21:59:24.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:23 smithi194 ceph-mon[29693]: pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:24.287 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.289+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c0062430 con 0x7fb8c0100740 2024-03-26T21:59:24.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.290+0000 7fb8a7fff700 1 -- 172.21.15.196:0/1670716499 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb8b0027030 con 0x7fb8c0100740 2024-03-26T21:59:24.288 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:24.288 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:24.290 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb8a8038760 msgr2=0x7fb8a803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb8a8038760 0x7fb8a803ac20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4006fd0 tx=0x7fb8b4006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 msgr2=0x7fb8c0074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0074a20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0006010 tx=0x7fb8b0009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 shutdown_connections 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb8a8038760 0x7fb8a803ac20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 --2- 172.21.15.196:0/1670716499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb8c0100740 0x7fb8c0074a20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.292+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 >> 172.21.15.196:0/1670716499 conn(0x7fb8c00fbd00 msgr2=0x7fb8c00fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.293+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 shutdown_connections 2024-03-26T21:59:24.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:24.293+0000 7fb8c55ce700 1 -- 172.21.15.196:0/1670716499 wait complete. 2024-03-26T21:59:24.292 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:25.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:25 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1670716499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:25.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:25 smithi194 ceph-mon[29693]: pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:25.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:25.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:26.121 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:26.167 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 -- 172.21.15.196:0/120865479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 msgr2=0x7fe78c0fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 --2- 172.21.15.196:0/120865479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c0fee10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009910 tx=0x7fe77c009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 -- 172.21.15.196:0/120865479 shutdown_connections 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 --2- 172.21.15.196:0/120865479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c0fee10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 -- 172.21.15.196:0/120865479 >> 172.21.15.196:0/120865479 conn(0x7fe78c0fa330 msgr2=0x7fe78c0fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 -- 172.21.15.196:0/120865479 shutdown_connections 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 -- 172.21.15.196:0/120865479 wait complete. 2024-03-26T21:59:27.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.666+0000 7fe790dc4700 1 Processor -- start 2024-03-26T21:59:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe790dc4700 1 -- start start 2024-03-26T21:59:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe790dc4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe790dc4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c196120 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe78a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe78a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36364/0 (socket says 172.21.15.196:36364) 2024-03-26T21:59:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe78a59c700 1 -- 172.21.15.196:0/3486606163 learned_addr learned my addr 172.21.15.196:0/3486606163 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:27.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.667+0000 7fe78a59c700 1 -- 172.21.15.196:0/3486606163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c0095f0 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.668+0000 7fe78a59c700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c195be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe77c006010 tx=0x7fe77c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.668+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe77c01d070 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.668+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c196320 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.668+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe77c005dc0 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.668+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c1967c0 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.668+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe77c022e50 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.669+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe78c18f570 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.669+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe77c022470 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.669+0000 7fe77b7fe700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe774038770 0x7fe77403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.669+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe77c04c650 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.669+0000 7fe789d9b700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe774038770 0x7fe77403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.670+0000 7fe789d9b700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe774038770 0x7fe77403ac30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe780006fd0 tx=0x7fe780006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:27.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.674+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe77c00baf0 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.944 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.946+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe78c0ff520 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.945 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.947+0000 7fe77b7fe700 1 -- 172.21.15.196:0/3486606163 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe78c0ff520 con 0x7fe78c0fe9f0 2024-03-26T21:59:27.945 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:27.945 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:27.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe774038770 msgr2=0x7fe77403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:27.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe774038770 0x7fe77403ac30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe780006fd0 tx=0x7fe780006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:27.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 msgr2=0x7fe78c195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:27.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c195be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe77c006010 tx=0x7fe77c009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 shutdown_connections 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe774038770 0x7fe77403ac30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 --2- 172.21.15.196:0/3486606163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe78c0fe9f0 0x7fe78c195be0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 >> 172.21.15.196:0/3486606163 conn(0x7fe78c0fa330 msgr2=0x7fe78c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 shutdown_connections 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:27.949+0000 7fe790dc4700 1 -- 172.21.15.196:0/3486606163 wait complete. 2024-03-26T21:59:27.948 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:28.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:27 smithi194 ceph-mon[29693]: pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:29.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:28 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3486606163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:29.556 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:29.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:29.757 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:29.802 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:30.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:29 smithi194 ceph-mon[29693]: pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:31.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 -- 172.21.15.196:0/1650479421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 msgr2=0x7f59f4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:31.291 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 --2- 172.21.15.196:0/1650479421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4100b80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009910 tx=0x7f59e4009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:31.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 -- 172.21.15.196:0/1650479421 shutdown_connections 2024-03-26T21:59:31.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 --2- 172.21.15.196:0/1650479421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4100b80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:31.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 -- 172.21.15.196:0/1650479421 >> 172.21.15.196:0/1650479421 conn(0x7f59f40fbd40 msgr2=0x7f59f40fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:31.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 -- 172.21.15.196:0/1650479421 shutdown_connections 2024-03-26T21:59:31.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.293+0000 7f59f93c0700 1 -- 172.21.15.196:0/1650479421 wait complete. 2024-03-26T21:59:31.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.294+0000 7f59f93c0700 1 Processor -- start 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.294+0000 7f59f93c0700 1 -- start start 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.294+0000 7f59f93c0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.294+0000 7f59f93c0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4196140 con 0x7f59f4100760 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.294+0000 7f59f2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.294+0000 7f59f2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36390/0 (socket says 172.21.15.196:36390) 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f2ffd700 1 -- 172.21.15.196:0/2463904536 learned_addr learned my addr 172.21.15.196:0/2463904536 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f2ffd700 1 -- 172.21.15.196:0/2463904536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40095f0 con 0x7f59f4100760 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f2ffd700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4195c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e401d070 con 0x7f59f4100760 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f4196340 con 0x7f59f4100760 2024-03-26T21:59:31.294 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59e4005dc0 con 0x7f59f4100760 2024-03-26T21:59:31.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e4022e50 con 0x7f59f4100760 2024-03-26T21:59:31.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.295+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f41967e0 con 0x7f59f4100760 2024-03-26T21:59:31.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.296+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f418f590 con 0x7f59f4100760 2024-03-26T21:59:31.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.296+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f59e4022470 con 0x7f59f4100760 2024-03-26T21:59:31.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.296+0000 7f59f0ff9700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f59e0038770 0x7f59e003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:31.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.296+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f59e404c800 con 0x7f59f4100760 2024-03-26T21:59:31.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.297+0000 7f59ebfff700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f59e0038770 0x7f59e003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:31.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.297+0000 7f59ebfff700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f59e0038770 0x7f59e003ac30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59dc006fd0 tx=0x7f59dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:31.299 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.301+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f59e400b960 con 0x7f59f4100760 2024-03-26T21:59:31.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.569+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59f40623e0 con 0x7f59f4100760 2024-03-26T21:59:31.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.569+0000 7f59f0ff9700 1 -- 172.21.15.196:0/2463904536 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f59f40623e0 con 0x7f59f4100760 2024-03-26T21:59:31.567 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:31.568 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:31.569 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f59e0038770 msgr2=0x7f59e003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f59e0038770 0x7f59e003ac30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59dc006fd0 tx=0x7f59dc006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 msgr2=0x7f59f4195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4195c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e4009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 shutdown_connections 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f59e0038770 0x7f59e003ac30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 --2- 172.21.15.196:0/2463904536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59f4100760 0x7f59f4195c00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 >> 172.21.15.196:0/2463904536 conn(0x7f59f40fbd40 msgr2=0x7f59f40fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.571+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 shutdown_connections 2024-03-26T21:59:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:31.572+0000 7f59f93c0700 1 -- 172.21.15.196:0/2463904536 wait complete. 2024-03-26T21:59:31.571 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:32.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:31 smithi194 ceph-mon[29693]: pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:32.187 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:31 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2463904536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:33.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:33.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:33.480 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:33.527 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:34.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:33 smithi194 ceph-mon[29693]: pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:34.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 -- 172.21.15.196:0/1405665404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 msgr2=0x7f9dd0100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:34.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 --2- 172.21.15.196:0/1405665404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0100b60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b0a0 tx=0x7f9dc800b3b0 comp rx=0 tx=0).stop 2024-03-26T21:59:34.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 -- 172.21.15.196:0/1405665404 shutdown_connections 2024-03-26T21:59:34.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 --2- 172.21.15.196:0/1405665404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0100b60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:34.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 -- 172.21.15.196:0/1405665404 >> 172.21.15.196:0/1405665404 conn(0x7f9dd00fbd00 msgr2=0x7f9dd00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:34.954 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 -- 172.21.15.196:0/1405665404 shutdown_connections 2024-03-26T21:59:34.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.955+0000 7f9dd8119700 1 -- 172.21.15.196:0/1405665404 wait complete. 2024-03-26T21:59:34.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.956+0000 7f9dd8119700 1 Processor -- start 2024-03-26T21:59:34.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.956+0000 7f9dd8119700 1 -- start start 2024-03-26T21:59:34.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.956+0000 7f9dd8119700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:34.955 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.956+0000 7f9dd8119700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0198340 con 0x7f9dd0100740 2024-03-26T21:59:34.956 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.956+0000 7f9dd5eb5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:34.956 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dd5eb5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55244/0 (socket says 172.21.15.196:55244) 2024-03-26T21:59:34.956 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dd5eb5700 1 -- 172.21.15.196:0/331788700 learned_addr learned my addr 172.21.15.196:0/331788700 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:34.956 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dd5eb5700 1 -- 172.21.15.196:0/331788700 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8009d00 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dd5eb5700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0197e00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80062a0 tx=0x7f9dc800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc800e040 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0198540 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd01989e0 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dc8003d70 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.957+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc80268e0 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.958+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9dc801c400 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.958+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db40052f0 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.958+0000 7f9dc6ffd700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dbc0387c0 0x7f9dbc03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.958+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9dc804c900 con 0x7f9dd0100740 2024-03-26T21:59:34.957 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.959+0000 7f9dd56b4700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dbc0387c0 0x7f9dbc03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:34.958 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.959+0000 7f9dd56b4700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dbc0387c0 0x7f9dbc03ac80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0006fd0 tx=0x7f9dc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:34.962 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:34.964+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9dc8024950 con 0x7f9dd0100740 2024-03-26T21:59:35.229 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.231+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9db4005160 con 0x7f9dd0100740 2024-03-26T21:59:35.229 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.231+0000 7f9dc6ffd700 1 -- 172.21.15.196:0/331788700 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9dc801b360 con 0x7f9dd0100740 2024-03-26T21:59:35.229 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:35.230 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dbc0387c0 msgr2=0x7f9dbc03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dbc0387c0 0x7f9dbc03ac80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0006fd0 tx=0x7f9dc0006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 msgr2=0x7f9dd0197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0197e00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80062a0 tx=0x7f9dc800bd40 comp rx=0 tx=0).stop 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 shutdown_connections 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dbc0387c0 0x7f9dbc03ac80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 --2- 172.21.15.196:0/331788700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd0100740 0x7f9dd0197e00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.233+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 >> 172.21.15.196:0/331788700 conn(0x7f9dd00fbd00 msgr2=0x7f9dd00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.234+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 shutdown_connections 2024-03-26T21:59:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:35.234+0000 7f9dd8119700 1 -- 172.21.15.196:0/331788700 wait complete. 2024-03-26T21:59:35.233 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:35.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:35 smithi194 ceph-mon[29693]: pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:36.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:36 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/331788700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:36.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:36.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:37.059 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:37.104 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:37.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:37 smithi194 ceph-mon[29693]: pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.620+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/1426999261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 msgr2=0x7fd8d01035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.620+0000 7fd8d4fb5700 1 --2- 172.21.15.196:0/1426999261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d01035a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800b0a0 tx=0x7fd8c800b3b0 comp rx=0 tx=0).stop 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.620+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/1426999261 shutdown_connections 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.620+0000 7fd8d4fb5700 1 --2- 172.21.15.196:0/1426999261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d01035a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.620+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/1426999261 >> 172.21.15.196:0/1426999261 conn(0x7fd8d00fab80 msgr2=0x7fd8d00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.620+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/1426999261 shutdown_connections 2024-03-26T21:59:38.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.621+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/1426999261 wait complete. 2024-03-26T21:59:38.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.621+0000 7fd8d4fb5700 1 Processor -- start 2024-03-26T21:59:38.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.621+0000 7fd8d4fb5700 1 -- start start 2024-03-26T21:59:38.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8d4fb5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d0195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:38.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8d4fb5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d01960a0 con 0x7fd8d01011b0 2024-03-26T21:59:38.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8ce59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d0195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:38.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8ce59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d0195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55260/0 (socket says 172.21.15.196:55260) 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8ce59c700 1 -- 172.21.15.196:0/736171079 learned_addr learned my addr 172.21.15.196:0/736171079 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8ce59c700 1 -- 172.21.15.196:0/736171079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c8009d00 con 0x7fd8d01011b0 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.622+0000 7fd8ce59c700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d0195b60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8015040 tx=0x7fd8c800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8c800e040 con 0x7fd8d01011b0 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8c8003d70 con 0x7fd8d01011b0 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d01962a0 con 0x7fd8d01011b0 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8c801c9e0 con 0x7fd8d01011b0 2024-03-26T21:59:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d0196740 con 0x7fd8d01011b0 2024-03-26T21:59:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fd8c8012640 con 0x7fd8d01011b0 2024-03-26T21:59:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.623+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d018f550 con 0x7fd8d01011b0 2024-03-26T21:59:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.624+0000 7fd8bf7fe700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd8b8038810 0x7fd8b803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.624+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd8c804c8f0 con 0x7fd8d01011b0 2024-03-26T21:59:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.624+0000 7fd8cdd9b700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd8b8038810 0x7fd8b803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.624+0000 7fd8cdd9b700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd8b8038810 0x7fd8b803acd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4006fd0 tx=0x7fd8c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:38.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.629+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd8c801a590 con 0x7fd8d01011b0 2024-03-26T21:59:38.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.898+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8d00623e0 con 0x7fd8d01011b0 2024-03-26T21:59:38.897 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.899+0000 7fd8bf7fe700 1 -- 172.21.15.196:0/736171079 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd8d00623e0 con 0x7fd8d01011b0 2024-03-26T21:59:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:38.897 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:38.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd8b8038810 msgr2=0x7fd8b803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:38.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd8b8038810 0x7fd8b803acd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4006fd0 tx=0x7fd8c4006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:38.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 msgr2=0x7fd8d0195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:38.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d0195b60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8015040 tx=0x7fd8c800bd40 comp rx=0 tx=0).stop 2024-03-26T21:59:38.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 shutdown_connections 2024-03-26T21:59:38.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd8b8038810 0x7fd8b803acd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:38.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 --2- 172.21.15.196:0/736171079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8d01011b0 0x7fd8d0195b60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:38.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 >> 172.21.15.196:0/736171079 conn(0x7fd8d00fab80 msgr2=0x7fd8d00fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:38.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 shutdown_connections 2024-03-26T21:59:38.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:38.901+0000 7fd8d4fb5700 1 -- 172.21.15.196:0/736171079 wait complete. 2024-03-26T21:59:38.900 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:40.117 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:39 smithi194 ceph-mon[29693]: pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:40.117 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:39 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/736171079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:40.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:40.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:40.682 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:40.728 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:42.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:41 smithi194 ceph-mon[29693]: pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:42.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 -- 172.21.15.196:0/619228077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 msgr2=0x7f54500ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:42.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 --2- 172.21.15.196:0/619228077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f54500ffa50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5440009910 tx=0x7f5440009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:42.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 -- 172.21.15.196:0/619228077 shutdown_connections 2024-03-26T21:59:42.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 --2- 172.21.15.196:0/619228077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f54500ffa50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:42.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 -- 172.21.15.196:0/619228077 >> 172.21.15.196:0/619228077 conn(0x7f54500fabd0 msgr2=0x7f54500fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:42.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 -- 172.21.15.196:0/619228077 shutdown_connections 2024-03-26T21:59:42.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.210+0000 7f5454ec9700 1 -- 172.21.15.196:0/619228077 wait complete. 2024-03-26T21:59:42.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f5454ec9700 1 Processor -- start 2024-03-26T21:59:42.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f5454ec9700 1 -- start start 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f5454ec9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f5450195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f5454ec9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5450196180 con 0x7f54500ff630 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f544e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f5450195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f544e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f5450195c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55282/0 (socket says 172.21.15.196:55282) 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.211+0000 7f544e59c700 1 -- 172.21.15.196:0/1789601098 learned_addr learned my addr 172.21.15.196:0/1789601098 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f544e59c700 1 -- 172.21.15.196:0/1789601098 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400095f0 con 0x7f54500ff630 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f544e59c700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f5450195c40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5440006010 tx=0x7f5440009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f544001d070 con 0x7f54500ff630 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5440005dc0 con 0x7f54500ff630 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5450196380 con 0x7f54500ff630 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5440022e90 con 0x7f54500ff630 2024-03-26T21:59:42.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.212+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450196820 con 0x7f54500ff630 2024-03-26T21:59:42.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.213+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545018f740 con 0x7f54500ff630 2024-03-26T21:59:42.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.213+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f5440022470 con 0x7f54500ff630 2024-03-26T21:59:42.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.213+0000 7f543f7fe700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5438038760 0x7f543803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:42.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.213+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f544004c870 con 0x7f54500ff630 2024-03-26T21:59:42.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.213+0000 7f544dd9b700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5438038760 0x7f543803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:42.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.214+0000 7f544dd9b700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5438038760 0x7f543803ac20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5444006fd0 tx=0x7f5444006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:42.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.218+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f544000b960 con 0x7f54500ff630 2024-03-26T21:59:42.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.486+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54500623e0 con 0x7f54500ff630 2024-03-26T21:59:42.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.487+0000 7f543f7fe700 1 -- 172.21.15.196:0/1789601098 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f54500623e0 con 0x7f54500ff630 2024-03-26T21:59:42.485 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:42.485 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5438038760 msgr2=0x7f543803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5438038760 0x7f543803ac20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5444006fd0 tx=0x7f5444006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 msgr2=0x7f5450195c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f5450195c40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5440006010 tx=0x7f5440009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 shutdown_connections 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5438038760 0x7f543803ac20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 --2- 172.21.15.196:0/1789601098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54500ff630 0x7f5450195c40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 >> 172.21.15.196:0/1789601098 conn(0x7f54500fabd0 msgr2=0x7f5450068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 shutdown_connections 2024-03-26T21:59:42.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:42.489+0000 7f5454ec9700 1 -- 172.21.15.196:0/1789601098 wait complete. 2024-03-26T21:59:42.488 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:43.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:42 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1789601098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:44.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:44.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:44.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:43 smithi194 ceph-mon[29693]: pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:44.221 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:44.268 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:45.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:45 smithi194 ceph-mon[29693]: pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:45.739 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 -- 172.21.15.196:0/1044104366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 msgr2=0x7f8870100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:45.739 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 --2- 172.21.15.196:0/1044104366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870100b40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8864009910 tx=0x7f8864009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:45.739 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 -- 172.21.15.196:0/1044104366 shutdown_connections 2024-03-26T21:59:45.739 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 --2- 172.21.15.196:0/1044104366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870100b40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:45.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 -- 172.21.15.196:0/1044104366 >> 172.21.15.196:0/1044104366 conn(0x7f88700fbcc0 msgr2=0x7f88700fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:45.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 -- 172.21.15.196:0/1044104366 shutdown_connections 2024-03-26T21:59:45.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.742+0000 7f8874a4f700 1 -- 172.21.15.196:0/1044104366 wait complete. 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f8874a4f700 1 Processor -- start 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f8874a4f700 1 -- start start 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f8874a4f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f8874a4f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870074f60 con 0x7f8870100720 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f886e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f886e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58866/0 (socket says 172.21.15.196:58866) 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f886e59c700 1 -- 172.21.15.196:0/2263021949 learned_addr learned my addr 172.21.15.196:0/2263021949 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:45.741 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f886e59c700 1 -- 172.21.15.196:0/2263021949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88640095f0 con 0x7f8870100720 2024-03-26T21:59:45.742 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.743+0000 7f886e59c700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870074a20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8864006010 tx=0x7f8864009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:45.742 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.744+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886401d070 con 0x7f8870100720 2024-03-26T21:59:45.742 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.744+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700731a0 con 0x7f8870100720 2024-03-26T21:59:45.742 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.744+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8864005dc0 con 0x7f8870100720 2024-03-26T21:59:45.742 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.744+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8864022e90 con 0x7f8870100720 2024-03-26T21:59:45.742 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.744+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8870073640 con 0x7f8870100720 2024-03-26T21:59:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.745+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f886400b960 con 0x7f8870100720 2024-03-26T21:59:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.745+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870106e80 con 0x7f8870100720 2024-03-26T21:59:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.745+0000 7f885f7fe700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8858038760 0x7f885803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.745+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f886404c8c0 con 0x7f8870100720 2024-03-26T21:59:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.745+0000 7f886dd9b700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8858038760 0x7f885803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.745+0000 7f886dd9b700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8858038760 0x7f885803ac20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8860006fd0 tx=0x7f8860006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:45.747 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:45.750+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8864022750 con 0x7f8870100720 2024-03-26T21:59:46.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.020+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88700623e0 con 0x7f8870100720 2024-03-26T21:59:46.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.021+0000 7f885f7fe700 1 -- 172.21.15.196:0/2263021949 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8864027020 con 0x7f8870100720 2024-03-26T21:59:46.019 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:46.019 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:46.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8858038760 msgr2=0x7f885803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:46.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8858038760 0x7f885803ac20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8860006fd0 tx=0x7f8860006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:46.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 msgr2=0x7f8870074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:46.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870074a20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8864006010 tx=0x7f8864009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:46.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 shutdown_connections 2024-03-26T21:59:46.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8858038760 0x7f885803ac20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:46.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 --2- 172.21.15.196:0/2263021949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8870100720 0x7f8870074a20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:46.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 >> 172.21.15.196:0/2263021949 conn(0x7f88700fbcc0 msgr2=0x7f88700fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:46.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 shutdown_connections 2024-03-26T21:59:46.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:46.023+0000 7f8874a4f700 1 -- 172.21.15.196:0/2263021949 wait complete. 2024-03-26T21:59:46.022 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:46.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:46 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2263021949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:47.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:47 smithi194 ceph-mon[29693]: pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:47.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:47.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:47.825 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:47.870 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 -- 172.21.15.196:0/2095039942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 msgr2=0x7f48800ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 --2- 172.21.15.196:0/2095039942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f48800ffa40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4870009910 tx=0x7f4870009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 -- 172.21.15.196:0/2095039942 shutdown_connections 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 --2- 172.21.15.196:0/2095039942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f48800ffa40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 -- 172.21.15.196:0/2095039942 >> 172.21.15.196:0/2095039942 conn(0x7f48800fabc0 msgr2=0x7f48800fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 -- 172.21.15.196:0/2095039942 shutdown_connections 2024-03-26T21:59:49.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.251+0000 7f4885fa1700 1 -- 172.21.15.196:0/2095039942 wait complete. 2024-03-26T21:59:49.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f4885fa1700 1 Processor -- start 2024-03-26T21:59:49.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f4885fa1700 1 -- start start 2024-03-26T21:59:49.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f4885fa1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f4880069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:49.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f4885fa1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4880069a40 con 0x7f48800ff620 2024-03-26T21:59:49.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f487f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f4880069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:49.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f487f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f4880069500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58888/0 (socket says 172.21.15.196:58888) 2024-03-26T21:59:49.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.252+0000 7f487f7fe700 1 -- 172.21.15.196:0/1879398406 learned_addr learned my addr 172.21.15.196:0/1879398406 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:49.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f487f7fe700 1 -- 172.21.15.196:0/1879398406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700095f0 con 0x7f48800ff620 2024-03-26T21:59:49.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f487f7fe700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f4880069500 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4870006010 tx=0x7f4870009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:49.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f487001d070 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4870005dc0 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4880198900 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4870022e90 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.253+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4880198da0 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.254+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f487000f4b0 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.254+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488018e450 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.254+0000 7f487cff9700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f48680387c0 0x7f486803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.254+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f487004ca80 con 0x7f48800ff620 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.254+0000 7f487effd700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f48680387c0 0x7f486803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:49.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.255+0000 7f487effd700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f48680387c0 0x7f486803ac80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4874006fd0 tx=0x7f4874006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:49.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.259+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f487002a950 con 0x7f48800ff620 2024-03-26T21:59:49.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.530+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48800623e0 con 0x7f48800ff620 2024-03-26T21:59:49.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.531+0000 7f487cff9700 1 -- 172.21.15.196:0/1879398406 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f48800623e0 con 0x7f48800ff620 2024-03-26T21:59:49.529 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:49.529 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:49.531 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f48680387c0 msgr2=0x7f486803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:49.531 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f48680387c0 0x7f486803ac80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4874006fd0 tx=0x7f4874006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:49.531 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 msgr2=0x7f4880069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:49.531 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f4880069500 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4870006010 tx=0x7f4870009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:49.531 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 shutdown_connections 2024-03-26T21:59:49.532 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f48680387c0 0x7f486803ac80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:49.532 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 --2- 172.21.15.196:0/1879398406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48800ff620 0x7f4880069500 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:49.532 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 >> 172.21.15.196:0/1879398406 conn(0x7f48800fabc0 msgr2=0x7f48800fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:49.532 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 shutdown_connections 2024-03-26T21:59:49.532 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:49.533+0000 7f4885fa1700 1 -- 172.21.15.196:0/1879398406 wait complete. 2024-03-26T21:59:49.532 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:50.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:49 smithi194 ceph-mon[29693]: pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:50.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:49 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1879398406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:51.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:51.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:51.293 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:51.339 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:52.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:51 smithi194 ceph-mon[29693]: pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:52.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 -- 172.21.15.196:0/2142545269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 msgr2=0x7ffa500ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 --2- 172.21.15.196:0/2142545269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa500ffa70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa40009910 tx=0x7ffa40009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 -- 172.21.15.196:0/2142545269 shutdown_connections 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 --2- 172.21.15.196:0/2142545269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa500ffa70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 -- 172.21.15.196:0/2142545269 >> 172.21.15.196:0/2142545269 conn(0x7ffa500fabf0 msgr2=0x7ffa500fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 -- 172.21.15.196:0/2142545269 shutdown_connections 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.816+0000 7ffa548c2700 1 -- 172.21.15.196:0/2142545269 wait complete. 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa548c2700 1 Processor -- start 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa548c2700 1 -- start start 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa548c2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa50195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa548c2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa501960f0 con 0x7ffa500ff650 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa4e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa50195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa4e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa50195bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36916/0 (socket says 172.21.15.196:36916) 2024-03-26T21:59:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.817+0000 7ffa4e59c700 1 -- 172.21.15.196:0/3581939586 learned_addr learned my addr 172.21.15.196:0/3581939586 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:52.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa4e59c700 1 -- 172.21.15.196:0/3581939586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400095f0 con 0x7ffa500ff650 2024-03-26T21:59:52.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa4e59c700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa50195bb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffa40006010 tx=0x7ffa40009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa4001d070 con 0x7ffa500ff650 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa40005dc0 con 0x7ffa500ff650 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa501962f0 con 0x7ffa500ff650 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa40022e90 con 0x7ffa500ff650 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.818+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa50196790 con 0x7ffa500ff650 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.819+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5018f6d0 con 0x7ffa500ff650 2024-03-26T21:59:52.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.819+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ffa40022470 con 0x7ffa500ff650 2024-03-26T21:59:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.819+0000 7ffa3f7fe700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ffa38038760 0x7ffa3803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.819+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffa4004c7f0 con 0x7ffa500ff650 2024-03-26T21:59:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.819+0000 7ffa4dd9b700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ffa38038760 0x7ffa3803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.820+0000 7ffa4dd9b700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ffa38038760 0x7ffa3803ac20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa44006fd0 tx=0x7ffa44006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:52.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:52.824+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffa4000f4f0 con 0x7ffa500ff650 2024-03-26T21:59:53.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.097+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa500623e0 con 0x7ffa500ff650 2024-03-26T21:59:53.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.097+0000 7ffa3f7fe700 1 -- 172.21.15.196:0/3581939586 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ffa40027070 con 0x7ffa500ff650 2024-03-26T21:59:53.095 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:53.096 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ffa38038760 msgr2=0x7ffa3803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ffa38038760 0x7ffa3803ac20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa44006fd0 tx=0x7ffa44006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 msgr2=0x7ffa50195bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa50195bb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffa40006010 tx=0x7ffa40009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 shutdown_connections 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ffa38038760 0x7ffa3803ac20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 --2- 172.21.15.196:0/3581939586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa500ff650 0x7ffa50195bb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 >> 172.21.15.196:0/3581939586 conn(0x7ffa500fabf0 msgr2=0x7ffa50068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 shutdown_connections 2024-03-26T21:59:53.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:53.099+0000 7ffa548c2700 1 -- 172.21.15.196:0/3581939586 wait complete. 2024-03-26T21:59:53.099 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:54.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:53 smithi194 ceph-mon[29693]: pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:54.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:53 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3581939586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:54.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:54.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:54.896 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:54.942 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:55.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:55 smithi194 ceph-mon[29693]: pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:56.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.454+0000 7f3e08cfd700 1 -- 172.21.15.196:0/3786045961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 msgr2=0x7f3e040fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:56.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.454+0000 7f3e08cfd700 1 --2- 172.21.15.196:0/3786045961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e040fee30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009910 tx=0x7f3df8009c20 comp rx=0 tx=0).stop 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 -- 172.21.15.196:0/3786045961 shutdown_connections 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 --2- 172.21.15.196:0/3786045961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e040fee30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 -- 172.21.15.196:0/3786045961 >> 172.21.15.196:0/3786045961 conn(0x7f3e040fa370 msgr2=0x7f3e040fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 -- 172.21.15.196:0/3786045961 shutdown_connections 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 -- 172.21.15.196:0/3786045961 wait complete. 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 Processor -- start 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.455+0000 7f3e08cfd700 1 -- start start 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e08cfd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e04195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:56.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e08cfd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e041960b0 con 0x7f3e040fea10 2024-03-26T21:59:56.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e0259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e04195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:56.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e0259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e04195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36922/0 (socket says 172.21.15.196:36922) 2024-03-26T21:59:56.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e0259c700 1 -- 172.21.15.196:0/251092492 learned_addr learned my addr 172.21.15.196:0/251092492 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T21:59:56.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e0259c700 1 -- 172.21.15.196:0/251092492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df80095f0 con 0x7f3e040fea10 2024-03-26T21:59:56.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3e0259c700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e04195b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3df8006010 tx=0x7f3df8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.456+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3df801d070 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.457+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3df8005dc0 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.457+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e04196310 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.457+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3df8022e90 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.457+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e04198fe0 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.457+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0418f500 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.457+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3df8022470 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.458+0000 7f3df37fe700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dec0387c0 0x7f3dec03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.458+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3df804c850 con 0x7f3e040fea10 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.458+0000 7f3e01d9b700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dec0387c0 0x7f3dec03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T21:59:56.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.458+0000 7f3e01d9b700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dec0387c0 0x7f3dec03ac80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3df4006fd0 tx=0x7f3df4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T21:59:56.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.463+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3df800bce0 con 0x7f3e040fea10 2024-03-26T21:59:56.727 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.729+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e040623e0 con 0x7f3e040fea10 2024-03-26T21:59:56.727 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.730+0000 7f3df37fe700 1 -- 172.21.15.196:0/251092492 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3e040623e0 con 0x7f3e040fea10 2024-03-26T21:59:56.728 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T21:59:56.728 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dec0387c0 msgr2=0x7f3dec03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dec0387c0 0x7f3dec03ac80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3df4006fd0 tx=0x7f3df4006e40 comp rx=0 tx=0).stop 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 msgr2=0x7f3e04195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e04195b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3df8006010 tx=0x7f3df8009f80 comp rx=0 tx=0).stop 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 shutdown_connections 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dec0387c0 0x7f3dec03ac80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:56.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 --2- 172.21.15.196:0/251092492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e040fea10 0x7f3e04195b70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T21:59:56.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 >> 172.21.15.196:0/251092492 conn(0x7f3e040fa370 msgr2=0x7f3e040fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:59:56.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 shutdown_connections 2024-03-26T21:59:56.731 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T21:59:56.732+0000 7f3e08cfd700 1 -- 172.21.15.196:0/251092492 wait complete. 2024-03-26T21:59:56.731 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T21:59:58.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:57 smithi194 ceph-mon[29693]: pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:59:58.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:57 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/251092492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:59:58.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T21:59:58.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:59:58.489 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:59:58.534 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:00.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 -- 172.21.15.196:0/3922596294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 msgr2=0x7f5b44102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:00.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 --2- 172.21.15.196:0/3922596294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44102180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009910 tx=0x7f5b34009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:00.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 -- 172.21.15.196:0/3922596294 shutdown_connections 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 --2- 172.21.15.196:0/3922596294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44102180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 -- 172.21.15.196:0/3922596294 >> 172.21.15.196:0/3922596294 conn(0x7f5b440fa380 msgr2=0x7f5b440fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 -- 172.21.15.196:0/3922596294 shutdown_connections 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.081+0000 7f5b4957c700 1 -- 172.21.15.196:0/3922596294 wait complete. 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b4957c700 1 Processor -- start 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b4957c700 1 -- start start 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b4957c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b4957c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44101870 con 0x7f5b44101d60 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b42ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b42ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44101330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36944/0 (socket says 172.21.15.196:36944) 2024-03-26T22:00:00.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.082+0000 7f5b42ffd700 1 -- 172.21.15.196:0/3090854802 learned_addr learned my addr 172.21.15.196:0/3090854802 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b42ffd700 1 -- 172.21.15.196:0/3090854802 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b340095f0 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b42ffd700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44101330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5b34006010 tx=0x7f5b34009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b3401d070 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b440ffab0 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b34005dc0 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b34022e90 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.083+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b440fff50 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.084+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f5b34022470 con 0x7f5b44101d60 2024-03-26T22:00:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.084+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b44106ec0 con 0x7f5b44101d60 2024-03-26T22:00:00.084 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.084+0000 7f5b40ff9700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5b30038760 0x7f5b3003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:00.084 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.084+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b3404c7f0 con 0x7f5b44101d60 2024-03-26T22:00:00.084 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.084+0000 7f5b3bfff700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5b30038760 0x7f5b3003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:00.084 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.085+0000 7f5b3bfff700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5b30038760 0x7f5b3003ac20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c006fd0 tx=0x7f5b2c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:00.087 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.090+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5b3400b960 con 0x7f5b44101d60 2024-03-26T22:00:00.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 21:59:59 smithi194 ceph-mon[29693]: pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:00.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.359+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b440623e0 con 0x7f5b44101d60 2024-03-26T22:00:00.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.359+0000 7f5b40ff9700 1 -- 172.21.15.196:0/3090854802 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5b34027030 con 0x7f5b44101d60 2024-03-26T22:00:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:00.358 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5b30038760 msgr2=0x7f5b3003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5b30038760 0x7f5b3003ac20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c006fd0 tx=0x7f5b2c006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 msgr2=0x7f5b44101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44101330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5b34006010 tx=0x7f5b34009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 shutdown_connections 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5b30038760 0x7f5b3003ac20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 --2- 172.21.15.196:0/3090854802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b44101d60 0x7f5b44101330 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 >> 172.21.15.196:0/3090854802 conn(0x7f5b440fa380 msgr2=0x7f5b440fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 shutdown_connections 2024-03-26T22:00:00.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:00.361+0000 7f5b4957c700 1 -- 172.21.15.196:0/3090854802 wait complete. 2024-03-26T22:00:00.361 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:01.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:00 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3090854802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:01.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:01.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:02.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:01 smithi194 ceph-mon[29693]: pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:02.165 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:02.212 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.806+0000 7fea76fb5700 1 -- 172.21.15.196:0/587020623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 msgr2=0x7fea700ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.806+0000 7fea76fb5700 1 --2- 172.21.15.196:0/587020623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea700ffa50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea64009910 tx=0x7fea64009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 -- 172.21.15.196:0/587020623 shutdown_connections 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 --2- 172.21.15.196:0/587020623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea700ffa50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 -- 172.21.15.196:0/587020623 >> 172.21.15.196:0/587020623 conn(0x7fea700fabd0 msgr2=0x7fea700fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 -- 172.21.15.196:0/587020623 shutdown_connections 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 -- 172.21.15.196:0/587020623 wait complete. 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 Processor -- start 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 -- start start 2024-03-26T22:00:03.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea70195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:03.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.807+0000 7fea76fb5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea70196180 con 0x7fea700ff630 2024-03-26T22:00:03.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea74d51700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea70195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:03.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea74d51700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea70195c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56360/0 (socket says 172.21.15.196:56360) 2024-03-26T22:00:03.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea74d51700 1 -- 172.21.15.196:0/3576388324 learned_addr learned my addr 172.21.15.196:0/3576388324 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:03.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea74d51700 1 -- 172.21.15.196:0/3576388324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea640095f0 con 0x7fea700ff630 2024-03-26T22:00:03.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea74d51700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea70195c40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea64006010 tx=0x7fea64009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:03.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea6401d070 con 0x7fea700ff630 2024-03-26T22:00:03.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea70196380 con 0x7fea700ff630 2024-03-26T22:00:03.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea64005dc0 con 0x7fea700ff630 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.808+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea64022e90 con 0x7fea700ff630 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.809+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea70196820 con 0x7fea700ff630 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.809+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7018f740 con 0x7fea700ff630 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.809+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fea64022470 con 0x7fea700ff630 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.810+0000 7fea6dffb700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fea60038760 0x7fea6003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.810+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fea6404c7f0 con 0x7fea700ff630 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.810+0000 7fea6ffff700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fea60038760 0x7fea6003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.810+0000 7fea6ffff700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fea60038760 0x7fea6003ac20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006fd0 tx=0x7fea5c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:03.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:03.815+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea6400f4f0 con 0x7fea700ff630 2024-03-26T22:00:04.087 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.089+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea700623e0 con 0x7fea700ff630 2024-03-26T22:00:04.087 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.089+0000 7fea6dffb700 1 -- 172.21.15.196:0/3576388324 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fea700623e0 con 0x7fea700ff630 2024-03-26T22:00:04.087 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:04.088 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fea60038760 msgr2=0x7fea6003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fea60038760 0x7fea6003ac20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006fd0 tx=0x7fea5c006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 msgr2=0x7fea70195c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea70195c40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea64006010 tx=0x7fea64009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 shutdown_connections 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fea60038760 0x7fea6003ac20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 --2- 172.21.15.196:0/3576388324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea700ff630 0x7fea70195c40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.091+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 >> 172.21.15.196:0/3576388324 conn(0x7fea700fabd0 msgr2=0x7fea70068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.092+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 shutdown_connections 2024-03-26T22:00:04.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:04.092+0000 7fea76fb5700 1 -- 172.21.15.196:0/3576388324 wait complete. 2024-03-26T22:00:04.091 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:04.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:03 smithi194 ceph-mon[29693]: pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:05.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:05 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3576388324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:05.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:05 smithi194 ceph-mon[29693]: pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:05.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:05.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:05.871 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:05.916 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 -- 172.21.15.196:0/3936274944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 msgr2=0x7f373c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 --2- 172.21.15.196:0/3936274944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0ffa40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3738009910 tx=0x7f3738009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 -- 172.21.15.196:0/3936274944 shutdown_connections 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 --2- 172.21.15.196:0/3936274944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0ffa40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 -- 172.21.15.196:0/3936274944 >> 172.21.15.196:0/3936274944 conn(0x7f373c0fab80 msgr2=0x7f373c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 -- 172.21.15.196:0/3936274944 shutdown_connections 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.374+0000 7f3743599700 1 -- 172.21.15.196:0/3936274944 wait complete. 2024-03-26T22:00:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.375+0000 7f3743599700 1 Processor -- start 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.375+0000 7f3743599700 1 -- start start 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.375+0000 7f3743599700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.375+0000 7f3743599700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c0fcf70 con 0x7f373c0ff620 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3741335700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3741335700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56378/0 (socket says 172.21.15.196:56378) 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3741335700 1 -- 172.21.15.196:0/2238838490 learned_addr learned my addr 172.21.15.196:0/2238838490 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3741335700 1 -- 172.21.15.196:0/2238838490 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380095f0 con 0x7f373c0ff620 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3741335700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0fca30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3738006010 tx=0x7f3738009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f373801d070 con 0x7f373c0ff620 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373c0fb1b0 con 0x7f373c0ff620 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3738005dc0 con 0x7f373c0ff620 2024-03-26T22:00:07.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3738022e90 con 0x7f373c0ff620 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.376+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373c0fb650 con 0x7f373c0ff620 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.377+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373c105d80 con 0x7f373c0ff620 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.377+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3738022470 con 0x7f373c0ff620 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.377+0000 7f37327fc700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3728038760 0x7f372803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.377+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f373804c920 con 0x7f373c0ff620 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.378+0000 7f3740b34700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3728038760 0x7f372803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.378+0000 7f3740b34700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3728038760 0x7f372803ac20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f372c006fd0 tx=0x7f372c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:07.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.383+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f373800f4f0 con 0x7f373c0ff620 2024-03-26T22:00:07.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.653+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f373c18f100 con 0x7f373c0ff620 2024-03-26T22:00:07.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.653+0000 7f37327fc700 1 -- 172.21.15.196:0/2238838490 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3738027030 con 0x7f373c0ff620 2024-03-26T22:00:07.652 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:07.652 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3728038760 msgr2=0x7f372803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3728038760 0x7f372803ac20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f372c006fd0 tx=0x7f372c006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 msgr2=0x7f373c0fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0fca30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3738006010 tx=0x7f3738009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 shutdown_connections 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3728038760 0x7f372803ac20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 --2- 172.21.15.196:0/2238838490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f373c0ff620 0x7f373c0fca30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:07.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 >> 172.21.15.196:0/2238838490 conn(0x7f373c0fab80 msgr2=0x7f373c0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:07.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 shutdown_connections 2024-03-26T22:00:07.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:07.656+0000 7f3743599700 1 -- 172.21.15.196:0/2238838490 wait complete. 2024-03-26T22:00:07.655 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:08.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:07 smithi194 ceph-mon[29693]: pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:08.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:07 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2238838490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:09.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:09.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:09.564 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:09.609 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:10.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:09 smithi194 ceph-mon[29693]: pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 -- 172.21.15.196:0/1270516450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 msgr2=0x7facac100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 --2- 172.21.15.196:0/1270516450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac100ad0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009910 tx=0x7fac9c009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 -- 172.21.15.196:0/1270516450 shutdown_connections 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 --2- 172.21.15.196:0/1270516450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac100ad0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 -- 172.21.15.196:0/1270516450 >> 172.21.15.196:0/1270516450 conn(0x7facac0fbc70 msgr2=0x7facac0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 -- 172.21.15.196:0/1270516450 shutdown_connections 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 -- 172.21.15.196:0/1270516450 wait complete. 2024-03-26T22:00:11.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 Processor -- start 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.173+0000 7facb1cd7700 1 -- start start 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facb1cd7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facb1cd7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac196070 con 0x7facac1006b0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facab7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facab7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56390/0 (socket says 172.21.15.196:56390) 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facab7fe700 1 -- 172.21.15.196:0/2635482859 learned_addr learned my addr 172.21.15.196:0/2635482859 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facab7fe700 1 -- 172.21.15.196:0/2635482859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c0095f0 con 0x7facac1006b0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7facab7fe700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac195b30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac9c006010 tx=0x7fac9c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.174+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac9c01d070 con 0x7facac1006b0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.175+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac9c005dc0 con 0x7facac1006b0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.175+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac9c022e90 con 0x7facac1006b0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.175+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac196270 con 0x7facac1006b0 2024-03-26T22:00:11.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.175+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac196710 con 0x7facac1006b0 2024-03-26T22:00:11.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.175+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fac9c00f4f0 con 0x7facac1006b0 2024-03-26T22:00:11.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.176+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac18f4e0 con 0x7facac1006b0 2024-03-26T22:00:11.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.176+0000 7faca8ff9700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fac94038760 0x7fac9403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:11.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.176+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fac9c04cc90 con 0x7facac1006b0 2024-03-26T22:00:11.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.176+0000 7facaaffd700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fac94038760 0x7fac9403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:11.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.176+0000 7facaaffd700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fac94038760 0x7fac9403ac20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faca0006fd0 tx=0x7faca0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:11.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.181+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac9c017440 con 0x7facac1006b0 2024-03-26T22:00:11.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.450+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facac0623e0 con 0x7facac1006b0 2024-03-26T22:00:11.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.450+0000 7faca8ff9700 1 -- 172.21.15.196:0/2635482859 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7facac0623e0 con 0x7facac1006b0 2024-03-26T22:00:11.449 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:11.449 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fac94038760 msgr2=0x7fac9403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fac94038760 0x7fac9403ac20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faca0006fd0 tx=0x7faca0006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 msgr2=0x7facac195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac195b30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac9c006010 tx=0x7fac9c009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 shutdown_connections 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fac94038760 0x7fac9403ac20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 --2- 172.21.15.196:0/2635482859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facac1006b0 0x7facac195b30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 >> 172.21.15.196:0/2635482859 conn(0x7facac0fbc70 msgr2=0x7facac0fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.452+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 shutdown_connections 2024-03-26T22:00:11.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:11.453+0000 7facb1cd7700 1 -- 172.21.15.196:0/2635482859 wait complete. 2024-03-26T22:00:11.452 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:12.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:11 smithi194 ceph-mon[29693]: pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:12.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:11 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2635482859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:13.057 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:13.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:13.257 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:13.302 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:14.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:13 smithi194 ceph-mon[29693]: pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:14.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/1392867680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 msgr2=0x7f4c14100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:14.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 --2- 172.21.15.196:0/1392867680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14100b40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00b0a0 tx=0x7f4c0c00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:00:14.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/1392867680 shutdown_connections 2024-03-26T22:00:14.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 --2- 172.21.15.196:0/1392867680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14100b40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:14.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/1392867680 >> 172.21.15.196:0/1392867680 conn(0x7f4c140fbcc0 msgr2=0x7f4c140fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:14.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/1392867680 shutdown_connections 2024-03-26T22:00:14.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.668+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/1392867680 wait complete. 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.669+0000 7f4c1bbd8700 1 Processor -- start 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.669+0000 7f4c1bbd8700 1 -- start start 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.669+0000 7f4c1bbd8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.669+0000 7f4c1bbd8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c14196100 con 0x7f4c14100720 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c19974700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c19974700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:54672/0 (socket says 172.21.15.196:54672) 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c19974700 1 -- 172.21.15.196:0/2933125768 learned_addr learned my addr 172.21.15.196:0/2933125768 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c19974700 1 -- 172.21.15.196:0/2933125768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c009d00 con 0x7f4c14100720 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c19974700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14195bc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0062a0 tx=0x7f4c0c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c0c00e040 con 0x7f4c14100720 2024-03-26T22:00:14.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c14196300 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c0c003d70 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c0c01c890 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.670+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c141967a0 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.671+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f4c0c01cb50 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.671+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1418f550 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.672+0000 7f4c06ffd700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4c000387c0 0x7f4c0003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.672+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4c0c04c9a0 con 0x7f4c14100720 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.672+0000 7f4c19173700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4c000387c0 0x7f4c0003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:14.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.672+0000 7f4c19173700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4c000387c0 0x7f4c0003ac80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c10006fd0 tx=0x7f4c10006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:14.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.677+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4c0c01a3b0 con 0x7f4c14100720 2024-03-26T22:00:14.943 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.945+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c140623e0 con 0x7f4c14100720 2024-03-26T22:00:14.944 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.946+0000 7f4c06ffd700 1 -- 172.21.15.196:0/2933125768 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4c0c01a3b0 con 0x7f4c14100720 2024-03-26T22:00:14.944 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:14.946 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4c000387c0 msgr2=0x7f4c0003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4c000387c0 0x7f4c0003ac80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c10006fd0 tx=0x7f4c10006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 msgr2=0x7f4c14195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14195bc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0062a0 tx=0x7f4c0c00bd40 comp rx=0 tx=0).stop 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 shutdown_connections 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4c000387c0 0x7f4c0003ac80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 --2- 172.21.15.196:0/2933125768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c14100720 0x7f4c14195bc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 >> 172.21.15.196:0/2933125768 conn(0x7f4c140fbcc0 msgr2=0x7f4c140fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 shutdown_connections 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:14.948+0000 7f4c1bbd8700 1 -- 172.21.15.196:0/2933125768 wait complete. 2024-03-26T22:00:14.947 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:15.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:15 smithi194 ceph-mon[29693]: pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:15.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:15 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2933125768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:16.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:16.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:16.747 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:16.793 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:18.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:17 smithi194 ceph-mon[29693]: pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:18.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 -- 172.21.15.196:0/2106920198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 msgr2=0x7f80d40ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:18.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 --2- 172.21.15.196:0/2106920198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d40ffa40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f80cc009910 tx=0x7f80cc009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:18.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 -- 172.21.15.196:0/2106920198 shutdown_connections 2024-03-26T22:00:18.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 --2- 172.21.15.196:0/2106920198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d40ffa40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 -- 172.21.15.196:0/2106920198 >> 172.21.15.196:0/2106920198 conn(0x7f80d40fab80 msgr2=0x7f80d40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 -- 172.21.15.196:0/2106920198 shutdown_connections 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.355+0000 7f80db4dc700 1 -- 172.21.15.196:0/2106920198 wait complete. 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80db4dc700 1 Processor -- start 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80db4dc700 1 -- start start 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80db4dc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d4195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80db4dc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d4196130 con 0x7f80d40ff620 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80d9278700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d4195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80d9278700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d4195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:54678/0 (socket says 172.21.15.196:54678) 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.356+0000 7f80d9278700 1 -- 172.21.15.196:0/1679755781 learned_addr learned my addr 172.21.15.196:0/1679755781 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80d9278700 1 -- 172.21.15.196:0/1679755781 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc0095f0 con 0x7f80d40ff620 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80d9278700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d4195bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f80cc006010 tx=0x7f80cc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:18.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80cc01d070 con 0x7f80d40ff620 2024-03-26T22:00:18.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d4196330 con 0x7f80d40ff620 2024-03-26T22:00:18.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80cc005dc0 con 0x7f80d40ff620 2024-03-26T22:00:18.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80cc022e90 con 0x7f80d40ff620 2024-03-26T22:00:18.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.357+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d41967d0 con 0x7f80d40ff620 2024-03-26T22:00:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.358+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d418f720 con 0x7f80d40ff620 2024-03-26T22:00:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.358+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f80cc022470 con 0x7f80d40ff620 2024-03-26T22:00:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.358+0000 7f80ca7fc700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f80c0038760 0x7f80c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.358+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f80cc04c7d0 con 0x7f80d40ff620 2024-03-26T22:00:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.358+0000 7f80d8a77700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f80c0038760 0x7f80c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.359+0000 7f80d8a77700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f80c0038760 0x7f80c003ac20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80c4006fd0 tx=0x7f80c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:18.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.363+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f80cc00b960 con 0x7f80d40ff620 2024-03-26T22:00:18.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.635+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80d40623e0 con 0x7f80d40ff620 2024-03-26T22:00:18.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.636+0000 7f80ca7fc700 1 -- 172.21.15.196:0/1679755781 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f80d40623e0 con 0x7f80d40ff620 2024-03-26T22:00:18.634 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:18.635 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:18.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f80c0038760 msgr2=0x7f80c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:18.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f80c0038760 0x7f80c003ac20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80c4006fd0 tx=0x7f80c4006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:18.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 msgr2=0x7f80d4195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d4195bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f80cc006010 tx=0x7f80cc009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 shutdown_connections 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f80c0038760 0x7f80c003ac20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 --2- 172.21.15.196:0/1679755781 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80d40ff620 0x7f80d4195bf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 >> 172.21.15.196:0/1679755781 conn(0x7f80d40fab80 msgr2=0x7f80d40682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 shutdown_connections 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:18.638+0000 7f80db4dc700 1 -- 172.21.15.196:0/1679755781 wait complete. 2024-03-26T22:00:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:19.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:18 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1679755781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:20.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:19 smithi194 ceph-mon[29693]: pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:20.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:20.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:20.404 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:20.450 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1786933055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 msgr2=0x7f6d08100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 --2- 172.21.15.196:0/1786933055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08100b40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009910 tx=0x7f6cf8009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1786933055 shutdown_connections 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 --2- 172.21.15.196:0/1786933055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08100b40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1786933055 >> 172.21.15.196:0/1786933055 conn(0x7f6d080fbcc0 msgr2=0x7f6d080fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1786933055 shutdown_connections 2024-03-26T22:00:21.908 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.908+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1786933055 wait complete. 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d0e3dc700 1 Processor -- start 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d0e3dc700 1 -- start start 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d0e3dc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d0e3dc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08196100 con 0x7f6d08100720 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d07fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d07fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:54702/0 (socket says 172.21.15.196:54702) 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.909+0000 7f6d07fff700 1 -- 172.21.15.196:0/1772029357 learned_addr learned my addr 172.21.15.196:0/1772029357 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d07fff700 1 -- 172.21.15.196:0/1772029357 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80095f0 con 0x7f6d08100720 2024-03-26T22:00:21.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d07fff700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08195bc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006010 tx=0x7f6cf8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf801d070 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d08196300 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cf8005dc0 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d081967a0 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.910+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf8022e90 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.911+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0818f550 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.911+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6cf8022470 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.911+0000 7f6d057fa700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.911+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6cf804c8a0 con 0x7f6d08100720 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.911+0000 7f6d077fe700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.912+0000 7f6d077fe700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc006fd0 tx=0x7f6cfc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:21.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:21.916+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6cf800f570 con 0x7f6d08100720 2024-03-26T22:00:22.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:21 smithi194 ceph-mon[29693]: pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:22.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.186+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d080623e0 con 0x7f6d08100720 2024-03-26T22:00:22.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.187+0000 7f6d057fa700 1 -- 172.21.15.196:0/1772029357 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6cf8027030 con 0x7f6d08100720 2024-03-26T22:00:22.185 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:22.185 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:22.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 msgr2=0x7f6cf003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc006fd0 tx=0x7f6cfc006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 msgr2=0x7f6d08195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08195bc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006010 tx=0x7f6cf8009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 shutdown_connections 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6cf0038760 0x7f6cf003ac20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 --2- 172.21.15.196:0/1772029357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d08100720 0x7f6d08195bc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 >> 172.21.15.196:0/1772029357 conn(0x7f6d080fbcc0 msgr2=0x7f6d080fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:22.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 shutdown_connections 2024-03-26T22:00:22.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:22.189+0000 7f6d0e3dc700 1 -- 172.21.15.196:0/1772029357 wait complete. 2024-03-26T22:00:22.189 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:23.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:22 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1772029357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:23.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:23.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:23.995 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:24.040 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:24.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:23 smithi194 ceph-mon[29693]: pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:25.300 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:25 smithi194 ceph-mon[29693]: pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:25.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 -- 172.21.15.196:0/141842903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 msgr2=0x7fc6580ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:25.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 --2- 172.21.15.196:0/141842903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580ffa40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc654009910 tx=0x7fc654009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:25.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 -- 172.21.15.196:0/141842903 shutdown_connections 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 --2- 172.21.15.196:0/141842903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580ffa40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 -- 172.21.15.196:0/141842903 >> 172.21.15.196:0/141842903 conn(0x7fc6580fab80 msgr2=0x7fc6580fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 -- 172.21.15.196:0/141842903 shutdown_connections 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.564+0000 7fc66047d700 1 -- 172.21.15.196:0/141842903 wait complete. 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc66047d700 1 Processor -- start 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc66047d700 1 -- start start 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc66047d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc66047d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc658068e10 con 0x7fc6580ff620 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc65e219700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc65e219700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580688d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60168/0 (socket says 172.21.15.196:60168) 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc65e219700 1 -- 172.21.15.196:0/1956751025 learned_addr learned my addr 172.21.15.196:0/1956751025 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:25.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.565+0000 7fc65e219700 1 -- 172.21.15.196:0/1956751025 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540095f0 con 0x7fc6580ff620 2024-03-26T22:00:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc65e219700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580688d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc654006010 tx=0x7fc654009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65401d070 con 0x7fc6580ff620 2024-03-26T22:00:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc658198910 con 0x7fc6580ff620 2024-03-26T22:00:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc654005dc0 con 0x7fc6580ff620 2024-03-26T22:00:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc654022e90 con 0x7fc6580ff620 2024-03-26T22:00:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc658198d50 con 0x7fc6580ff620 2024-03-26T22:00:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.566+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65818e450 con 0x7fc6580ff620 2024-03-26T22:00:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.567+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fc654022470 con 0x7fc6580ff620 2024-03-26T22:00:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.567+0000 7fc64f7fe700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc644038760 0x7fc64403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.567+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc65404c7f0 con 0x7fc6580ff620 2024-03-26T22:00:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.567+0000 7fc65da18700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc644038760 0x7fc64403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.567+0000 7fc65da18700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc644038760 0x7fc64403ac20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc648006fd0 tx=0x7fc648006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:25.570 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.572+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc65400b960 con 0x7fc6580ff620 2024-03-26T22:00:25.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.841+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6580623e0 con 0x7fc6580ff620 2024-03-26T22:00:25.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.842+0000 7fc64f7fe700 1 -- 172.21.15.196:0/1956751025 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc654027030 con 0x7fc6580ff620 2024-03-26T22:00:25.841 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:25.841 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.844+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc644038760 msgr2=0x7fc64403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.844+0000 7fc66047d700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc644038760 0x7fc64403ac20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc648006fd0 tx=0x7fc648006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.844+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 msgr2=0x7fc6580688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.844+0000 7fc66047d700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580688d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc654006010 tx=0x7fc654009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.845+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 shutdown_connections 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.845+0000 7fc66047d700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc644038760 0x7fc64403ac20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.845+0000 7fc66047d700 1 --2- 172.21.15.196:0/1956751025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6580ff620 0x7fc6580688d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.845+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 >> 172.21.15.196:0/1956751025 conn(0x7fc6580fab80 msgr2=0x7fc6580fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.845+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 shutdown_connections 2024-03-26T22:00:25.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:25.845+0000 7fc66047d700 1 -- 172.21.15.196:0/1956751025 wait complete. 2024-03-26T22:00:25.844 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:26.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:26 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1956751025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:27.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:27 smithi194 ceph-mon[29693]: pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:27.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:27.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:27.706 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:27.752 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:29.322 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 -- 172.21.15.196:0/481922203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 msgr2=0x7f2290102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 --2- 172.21.15.196:0/481922203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290102150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2280009910 tx=0x7f2280009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 -- 172.21.15.196:0/481922203 shutdown_connections 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 --2- 172.21.15.196:0/481922203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290102150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 -- 172.21.15.196:0/481922203 >> 172.21.15.196:0/481922203 conn(0x7f22900fa310 msgr2=0x7f22900fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 -- 172.21.15.196:0/481922203 shutdown_connections 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.323+0000 7f2295679700 1 -- 172.21.15.196:0/481922203 wait complete. 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.324+0000 7f2295679700 1 Processor -- start 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.324+0000 7f2295679700 1 -- start start 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.324+0000 7f2295679700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:29.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.324+0000 7f2295679700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290196140 con 0x7f2290101d30 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60184/0 (socket says 172.21.15.196:60184) 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228effd700 1 -- 172.21.15.196:0/1439473990 learned_addr learned my addr 172.21.15.196:0/1439473990 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228effd700 1 -- 172.21.15.196:0/1439473990 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800095f0 con 0x7f2290101d30 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228effd700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290195c00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2280006010 tx=0x7f2280009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f228001d070 con 0x7f2290101d30 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2290196340 con 0x7f2290101d30 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2280005dc0 con 0x7f2290101d30 2024-03-26T22:00:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2280022e90 con 0x7f2290101d30 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.325+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22901967e0 con 0x7f2290101d30 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.326+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f228000f4f0 con 0x7f2290101d30 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.326+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229018f560 con 0x7f2290101d30 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.326+0000 7f228cff9700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f227c038760 0x7f227c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.327+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f228004cc10 con 0x7f2290101d30 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.327+0000 7f22867fc700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f227c038760 0x7f227c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:29.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.327+0000 7f22867fc700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f227c038760 0x7f227c03ac20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2278006fd0 tx=0x7f2278006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:29.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.333+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f228002a950 con 0x7f2290101d30 2024-03-26T22:00:29.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.602+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22900623e0 con 0x7f2290101d30 2024-03-26T22:00:29.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.603+0000 7f228cff9700 1 -- 172.21.15.196:0/1439473990 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f22900623e0 con 0x7f2290101d30 2024-03-26T22:00:29.601 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:29.602 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:29.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f227c038760 msgr2=0x7f227c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:29.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f227c038760 0x7f227c03ac20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2278006fd0 tx=0x7f2278006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:29.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 msgr2=0x7f2290195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:29.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290195c00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2280006010 tx=0x7f2280009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 shutdown_connections 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f227c038760 0x7f227c03ac20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 --2- 172.21.15.196:0/1439473990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2290101d30 0x7f2290195c00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 >> 172.21.15.196:0/1439473990 conn(0x7f22900fa310 msgr2=0x7f22900fae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 shutdown_connections 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:29.605+0000 7f2295679700 1 -- 172.21.15.196:0/1439473990 wait complete. 2024-03-26T22:00:29.604 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:30.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:29 smithi194 ceph-mon[29693]: pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:30.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:29 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1439473990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:31.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:31.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:31.393 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:31.440 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:32.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:31 smithi194 ceph-mon[29693]: pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:33.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 -- 172.21.15.196:0/3795054720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 msgr2=0x7fa4d0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:33.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 --2- 172.21.15.196:0/3795054720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0100b40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009910 tx=0x7fa4cc009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 -- 172.21.15.196:0/3795054720 shutdown_connections 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 --2- 172.21.15.196:0/3795054720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0100b40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 -- 172.21.15.196:0/3795054720 >> 172.21.15.196:0/3795054720 conn(0x7fa4d00fbcc0 msgr2=0x7fa4d00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 -- 172.21.15.196:0/3795054720 shutdown_connections 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.054+0000 7fa4d7dba700 1 -- 172.21.15.196:0/3795054720 wait complete. 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d7dba700 1 Processor -- start 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d7dba700 1 -- start start 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d7dba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d7dba700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0196100 con 0x7fa4d0100720 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d5b56700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:33.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d5b56700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58478/0 (socket says 172.21.15.196:58478) 2024-03-26T22:00:33.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d5b56700 1 -- 172.21.15.196:0/1308261078 learned_addr learned my addr 172.21.15.196:0/1308261078 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.055+0000 7fa4d5b56700 1 -- 172.21.15.196:0/1308261078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc0095f0 con 0x7fa4d0100720 2024-03-26T22:00:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.056+0000 7fa4d5b56700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0195bc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc006010 tx=0x7fa4cc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.056+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc01d070 con 0x7fa4d0100720 2024-03-26T22:00:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.056+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d0196300 con 0x7fa4d0100720 2024-03-26T22:00:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.056+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4cc005dc0 con 0x7fa4d0100720 2024-03-26T22:00:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.056+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc022e90 con 0x7fa4d0100720 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.056+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d01967a0 con 0x7fa4d0100720 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.057+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d018f550 con 0x7fa4d0100720 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.057+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa4cc022470 con 0x7fa4d0100720 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.057+0000 7fa4c6ffd700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4bc038760 0x7fa4bc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.057+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa4cc04c870 con 0x7fa4d0100720 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.057+0000 7fa4d5355700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4bc038760 0x7fa4bc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:33.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.058+0000 7fa4d5355700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4bc038760 0x7fa4bc03ac20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0006fd0 tx=0x7fa4c0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:33.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.062+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa4cc00b960 con 0x7fa4d0100720 2024-03-26T22:00:33.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.335+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4d00623e0 con 0x7fa4d0100720 2024-03-26T22:00:33.334 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.336+0000 7fa4c6ffd700 1 -- 172.21.15.196:0/1308261078 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa4d00623e0 con 0x7fa4d0100720 2024-03-26T22:00:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:33.335 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4bc038760 msgr2=0x7fa4bc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4bc038760 0x7fa4bc03ac20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0006fd0 tx=0x7fa4c0006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 msgr2=0x7fa4d0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0195bc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc006010 tx=0x7fa4cc009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 shutdown_connections 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa4bc038760 0x7fa4bc03ac20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:33.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 --2- 172.21.15.196:0/1308261078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4d0100720 0x7fa4d0195bc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:33.338 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.338+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 >> 172.21.15.196:0/1308261078 conn(0x7fa4d00fbcc0 msgr2=0x7fa4d00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:33.338 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.339+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 shutdown_connections 2024-03-26T22:00:33.338 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:33.339+0000 7fa4d7dba700 1 -- 172.21.15.196:0/1308261078 wait complete. 2024-03-26T22:00:33.338 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:33 smithi194 ceph-mon[29693]: pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:33 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1308261078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:34.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:34.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:35.096 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:35.141 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:35.314 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:35 smithi194 ceph-mon[29693]: pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:36.753 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/4008118607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 msgr2=0x7fb3c8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 --2- 172.21.15.196:0/4008118607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8100b40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009910 tx=0x7fb3b8009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/4008118607 shutdown_connections 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 --2- 172.21.15.196:0/4008118607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8100b40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/4008118607 >> 172.21.15.196:0/4008118607 conn(0x7fb3c80fbcc0 msgr2=0x7fb3c80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/4008118607 shutdown_connections 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.754+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/4008118607 wait complete. 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3cd5a0700 1 Processor -- start 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3cd5a0700 1 -- start start 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3cd5a0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3cd5a0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8074f60 con 0x7fb3c8100720 2024-03-26T22:00:36.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3c6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:36.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3c6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58496/0 (socket says 172.21.15.196:58496) 2024-03-26T22:00:36.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.755+0000 7fb3c6ffd700 1 -- 172.21.15.196:0/2370957085 learned_addr learned my addr 172.21.15.196:0/2370957085 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:36.756 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3c6ffd700 1 -- 172.21.15.196:0/2370957085 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b80095f0 con 0x7fb3c8100720 2024-03-26T22:00:36.756 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3c6ffd700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8074a20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8006010 tx=0x7fb3b8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:36.756 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3b801d070 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c80731a0 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3b8005dc0 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3b8022e90 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.756+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c8073640 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.757+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8106e80 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.757+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb3b8022470 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.757+0000 7fb3c4ff9700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb3b4038760 0x7fb3b403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.757+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb3b804c8b0 con 0x7fb3c8100720 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.757+0000 7fb3be7fc700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb3b4038760 0x7fb3b403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.758+0000 7fb3be7fc700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb3b4038760 0x7fb3b403ac20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0006fd0 tx=0x7fb3b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:36.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:36.762+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb3b800f4f0 con 0x7fb3c8100720 2024-03-26T22:00:37.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.033+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3c80623e0 con 0x7fb3c8100720 2024-03-26T22:00:37.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.034+0000 7fb3c4ff9700 1 -- 172.21.15.196:0/2370957085 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb3b8027070 con 0x7fb3c8100720 2024-03-26T22:00:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:37.032 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:37.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb3b4038760 msgr2=0x7fb3b403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb3b4038760 0x7fb3b403ac20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0006fd0 tx=0x7fb3b0006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 msgr2=0x7fb3c8074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8074a20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8006010 tx=0x7fb3b8009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 shutdown_connections 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb3b4038760 0x7fb3b403ac20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 --2- 172.21.15.196:0/2370957085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3c8100720 0x7fb3c8074a20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 >> 172.21.15.196:0/2370957085 conn(0x7fb3c80fbcc0 msgr2=0x7fb3c80fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 shutdown_connections 2024-03-26T22:00:37.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:37.036+0000 7fb3cd5a0700 1 -- 172.21.15.196:0/2370957085 wait complete. 2024-03-26T22:00:37.036 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:38.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:37 smithi194 ceph-mon[29693]: pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:38.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:37 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2370957085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:38.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:38.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:38.912 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:38.958 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:40.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:39 smithi194 ceph-mon[29693]: pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:40.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1473840036 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 msgr2=0x7f12d40ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 --2- 172.21.15.196:0/1473840036 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d40ffa40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009910 tx=0x7f12c4009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1473840036 shutdown_connections 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 --2- 172.21.15.196:0/1473840036 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d40ffa40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1473840036 >> 172.21.15.196:0/1473840036 conn(0x7f12d40fabc0 msgr2=0x7f12d40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1473840036 shutdown_connections 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.552+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1473840036 wait complete. 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d8cd1700 1 Processor -- start 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d8cd1700 1 -- start start 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d8cd1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d4195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:40.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d8cd1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4196110 con 0x7f12d40ff620 2024-03-26T22:00:40.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d4195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:40.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d4195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:58524/0 (socket says 172.21.15.196:58524) 2024-03-26T22:00:40.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.553+0000 7f12d259c700 1 -- 172.21.15.196:0/1523996459 learned_addr learned my addr 172.21.15.196:0/1523996459 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:40.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12d259c700 1 -- 172.21.15.196:0/1523996459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40095f0 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12d259c700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d4195bd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c401d070 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d4196310 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c4005dc0 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c4022e90 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.554+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d41967b0 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.555+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f12c400f530 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.555+0000 7f12cbfff700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f12c0038760 0x7f12c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.555+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d40623e0 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.555+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f12c404c7b0 con 0x7f12d40ff620 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.555+0000 7f12c9d9b700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f12c0038760 0x7f12c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.556+0000 7f12c9d9b700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f12c0038760 0x7f12c003ac20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f12bc006fd0 tx=0x7f12bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:40.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.561+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f12c4027070 con 0x7f12d40ff620 2024-03-26T22:00:40.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.831+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12d4105460 con 0x7f12d40ff620 2024-03-26T22:00:40.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.832+0000 7f12cbfff700 1 -- 172.21.15.196:0/1523996459 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f12c402a720 con 0x7f12d40ff620 2024-03-26T22:00:40.831 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:40.831 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:40.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f12c0038760 msgr2=0x7f12c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:40.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f12c0038760 0x7f12c003ac20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f12bc006fd0 tx=0x7f12bc006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 msgr2=0x7f12d4195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d4195bd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c4009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 shutdown_connections 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f12c0038760 0x7f12c003ac20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 --2- 172.21.15.196:0/1523996459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d40ff620 0x7f12d4195bd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 >> 172.21.15.196:0/1523996459 conn(0x7f12d40fabc0 msgr2=0x7f12d4068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 shutdown_connections 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:40.834+0000 7f12d8cd1700 1 -- 172.21.15.196:0/1523996459 wait complete. 2024-03-26T22:00:40.833 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:42.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:41 smithi194 ceph-mon[29693]: pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:42.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:41 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1523996459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:42.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:42.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:42.675 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:42.720 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:44.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:43 smithi194 ceph-mon[29693]: pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:44.297 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 -- 172.21.15.196:0/2403439626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 msgr2=0x7f7cb40ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:44.297 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 --2- 172.21.15.196:0/2403439626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb40ff440 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009910 tx=0x7f7ca4009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:44.297 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 -- 172.21.15.196:0/2403439626 shutdown_connections 2024-03-26T22:00:44.297 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 --2- 172.21.15.196:0/2403439626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb40ff440 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:44.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 -- 172.21.15.196:0/2403439626 >> 172.21.15.196:0/2403439626 conn(0x7f7cb40fab80 msgr2=0x7f7cb40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:44.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 -- 172.21.15.196:0/2403439626 shutdown_connections 2024-03-26T22:00:44.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.298+0000 7f7cb9d58700 1 -- 172.21.15.196:0/2403439626 wait complete. 2024-03-26T22:00:44.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb9d58700 1 Processor -- start 2024-03-26T22:00:44.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb9d58700 1 -- start start 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb9d58700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb4197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb9d58700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41982d0 con 0x7f7cb40ff020 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb4197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb4197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56404/0 (socket says 172.21.15.196:56404) 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.299+0000 7f7cb37fe700 1 -- 172.21.15.196:0/1362264369 learned_addr learned my addr 172.21.15.196:0/1362264369 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb37fe700 1 -- 172.21.15.196:0/1362264369 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40095f0 con 0x7f7cb40ff020 2024-03-26T22:00:44.300 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb37fe700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb4197d90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009ed0 tx=0x7f7ca4004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca401d070 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb41984d0 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ca400be20 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca400f460 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.300+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb4198970 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.301+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f7ca400b960 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.301+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb4191740 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.301+0000 7f7cb17fa700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7ca00387d0 0x7f7ca003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.301+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ca404c930 con 0x7f7cb40ff020 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.301+0000 7f7caaffd700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7ca00387d0 0x7f7ca003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:44.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.302+0000 7f7caaffd700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7ca00387d0 0x7f7ca003ac90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c006fd0 tx=0x7f7c9c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:44.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.306+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ca402a360 con 0x7f7cb40ff020 2024-03-26T22:00:44.574 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.575+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cb40623e0 con 0x7f7cb40ff020 2024-03-26T22:00:44.574 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.576+0000 7f7cb17fa700 1 -- 172.21.15.196:0/1362264369 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7cb40623e0 con 0x7f7cb40ff020 2024-03-26T22:00:44.575 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:44.575 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.578+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7ca00387d0 msgr2=0x7f7ca003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.578+0000 7f7cb9d58700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7ca00387d0 0x7f7ca003ac90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c006fd0 tx=0x7f7c9c006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.578+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 msgr2=0x7f7cb4197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.578+0000 7f7cb9d58700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb4197d90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009ed0 tx=0x7f7ca4004690 comp rx=0 tx=0).stop 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.579+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 shutdown_connections 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.579+0000 7f7cb9d58700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f7ca00387d0 0x7f7ca003ac90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.579+0000 7f7cb9d58700 1 --2- 172.21.15.196:0/1362264369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cb40ff020 0x7f7cb4197d90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:44.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.579+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 >> 172.21.15.196:0/1362264369 conn(0x7f7cb40fab80 msgr2=0x7f7cb40fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:44.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.579+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 shutdown_connections 2024-03-26T22:00:44.578 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:44.579+0000 7f7cb9d58700 1 -- 172.21.15.196:0/1362264369 wait complete. 2024-03-26T22:00:44.578 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:45.327 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:45 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1362264369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:45.327 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:45 smithi194 ceph-mon[29693]: pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:46.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:46.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:46.524 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:46.569 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:48.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 -- 172.21.15.196:0/1382800731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 msgr2=0x7f260c0ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:48.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 --2- 172.21.15.196:0/1382800731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c0ff3b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009910 tx=0x7f25fc009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:48.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 -- 172.21.15.196:0/1382800731 shutdown_connections 2024-03-26T22:00:48.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 --2- 172.21.15.196:0/1382800731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c0ff3b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:48.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 -- 172.21.15.196:0/1382800731 >> 172.21.15.196:0/1382800731 conn(0x7f260c0fabb0 msgr2=0x7f260c0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:48.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 -- 172.21.15.196:0/1382800731 shutdown_connections 2024-03-26T22:00:48.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.073+0000 7f26111dc700 1 -- 172.21.15.196:0/1382800731 wait complete. 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f26111dc700 1 Processor -- start 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f26111dc700 1 -- start start 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f26111dc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f26111dc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c1982e0 con 0x7f260c0fef90 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f260ad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f260ad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56432/0 (socket says 172.21.15.196:56432) 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.074+0000 7f260ad9d700 1 -- 172.21.15.196:0/1747259079 learned_addr learned my addr 172.21.15.196:0/1747259079 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:48.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f260ad9d700 1 -- 172.21.15.196:0/1747259079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0095f0 con 0x7f260c0fef90 2024-03-26T22:00:48.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f260ad9d700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c197da0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009ed0 tx=0x7f25fc004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:48.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25fc01d070 con 0x7f260c0fef90 2024-03-26T22:00:48.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c1984e0 con 0x7f260c0fef90 2024-03-26T22:00:48.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c198980 con 0x7f260c0fef90 2024-03-26T22:00:48.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25fc00be20 con 0x7f260c0fef90 2024-03-26T22:00:48.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.075+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25fc00f460 con 0x7f260c0fef90 2024-03-26T22:00:48.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.076+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f25fc00f5c0 con 0x7f260c0fef90 2024-03-26T22:00:48.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.076+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c191730 con 0x7f260c0fef90 2024-03-26T22:00:48.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.076+0000 7f25f3fff700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f40387d0 0x7f25f403ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:48.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.076+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f25fc04cc00 con 0x7f260c0fef90 2024-03-26T22:00:48.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.076+0000 7f260a59c700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f40387d0 0x7f25f403ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:48.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.077+0000 7f260a59c700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f40387d0 0x7f25f403ac90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2600006fd0 tx=0x7f2600006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:48.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.082+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f25fc021c20 con 0x7f260c0fef90 2024-03-26T22:00:48.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:47 smithi194 ceph-mon[29693]: pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:48.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.349+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f260c0623e0 con 0x7f260c0fef90 2024-03-26T22:00:48.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.350+0000 7f25f3fff700 1 -- 172.21.15.196:0/1747259079 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f260c0623e0 con 0x7f260c0fef90 2024-03-26T22:00:48.348 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:48.348 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:48.350 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f40387d0 msgr2=0x7f25f403ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f40387d0 0x7f25f403ac90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2600006fd0 tx=0x7f2600006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 msgr2=0x7f260c197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c197da0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009ed0 tx=0x7f25fc004690 comp rx=0 tx=0).stop 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 shutdown_connections 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f40387d0 0x7f25f403ac90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 --2- 172.21.15.196:0/1747259079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f260c0fef90 0x7f260c197da0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 >> 172.21.15.196:0/1747259079 conn(0x7f260c0fabb0 msgr2=0x7f260c0fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 shutdown_connections 2024-03-26T22:00:48.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:48.352+0000 7f26111dc700 1 -- 172.21.15.196:0/1747259079 wait complete. 2024-03-26T22:00:48.352 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:49.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:48 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1747259079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:49.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:49.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:50.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:49 smithi194 ceph-mon[29693]: pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:50.244 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:51.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 -- 172.21.15.196:0/707587721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 msgr2=0x7f41d00fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:51.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 --2- 172.21.15.196:0/707587721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d00fee10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009910 tx=0x7f41c0009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:51.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 -- 172.21.15.196:0/707587721 shutdown_connections 2024-03-26T22:00:51.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 --2- 172.21.15.196:0/707587721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d00fee10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:51.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 -- 172.21.15.196:0/707587721 >> 172.21.15.196:0/707587721 conn(0x7f41d00fa330 msgr2=0x7f41d00fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:51.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 -- 172.21.15.196:0/707587721 shutdown_connections 2024-03-26T22:00:51.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 -- 172.21.15.196:0/707587721 wait complete. 2024-03-26T22:00:51.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.861+0000 7f41d57a4700 1 Processor -- start 2024-03-26T22:00:51.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41d57a4700 1 -- start start 2024-03-26T22:00:51.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41d57a4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d0197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:51.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41d57a4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d0198340 con 0x7f41d00fe9f0 2024-03-26T22:00:51.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41ceffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d0197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:51.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41ceffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d0197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:56442/0 (socket says 172.21.15.196:56442) 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41ceffd700 1 -- 172.21.15.196:0/597877234 learned_addr learned my addr 172.21.15.196:0/597877234 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41ceffd700 1 -- 172.21.15.196:0/597877234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c00095f0 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.862+0000 7f41ceffd700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d0197e00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009ed0 tx=0x7f41c0004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.863+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c001d070 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.863+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d0198540 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.863+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41c000be20 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.863+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c000f460 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.863+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d01989e0 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.864+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f41c0022800 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.864+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d0191770 con 0x7f41d00fe9f0 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.864+0000 7f41b7fff700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f41b80387d0 0x7f41b803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:51.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.864+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f41c004c7b0 con 0x7f41d00fe9f0 2024-03-26T22:00:51.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.864+0000 7f41ce7fc700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f41b80387d0 0x7f41b803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:51.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.864+0000 7f41ce7fc700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f41b80387d0 0x7f41b803ac90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006fd0 tx=0x7f41c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:51.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:51.869+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f41c002b590 con 0x7f41d00fe9f0 2024-03-26T22:00:52.139 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.141+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41d00ff520 con 0x7f41d00fe9f0 2024-03-26T22:00:52.140 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.142+0000 7f41b7fff700 1 -- 172.21.15.196:0/597877234 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f41d00ff520 con 0x7f41d00fe9f0 2024-03-26T22:00:52.140 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:52.140 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:52.142 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f41b80387d0 msgr2=0x7f41b803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:52.142 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f41b80387d0 0x7f41b803ac90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006fd0 tx=0x7f41c4006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 msgr2=0x7f41d0197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d0197e00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009ed0 tx=0x7f41c0004690 comp rx=0 tx=0).stop 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 shutdown_connections 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f41b80387d0 0x7f41b803ac90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 --2- 172.21.15.196:0/597877234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41d00fe9f0 0x7f41d0197e00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 >> 172.21.15.196:0/597877234 conn(0x7f41d00fa330 msgr2=0x7f41d00fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 shutdown_connections 2024-03-26T22:00:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:52.144+0000 7f41d57a4700 1 -- 172.21.15.196:0/597877234 wait complete. 2024-03-26T22:00:52.144 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:52.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:51 smithi194 ceph-mon[29693]: pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:53.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:52 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/597877234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:53.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:53.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:53.911 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:53.957 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:54.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:53 smithi194 ceph-mon[29693]: pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:55.340 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:55 smithi194 ceph-mon[29693]: pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:55.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.602+0000 7f8f3973d700 1 -- 172.21.15.196:0/1391114923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 msgr2=0x7f8f34103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:55.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.602+0000 7f8f3973d700 1 --2- 172.21.15.196:0/1391114923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34103660 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009910 tx=0x7f8f24009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:55.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.602+0000 7f8f3973d700 1 -- 172.21.15.196:0/1391114923 shutdown_connections 2024-03-26T22:00:55.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.602+0000 7f8f3973d700 1 --2- 172.21.15.196:0/1391114923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34103660 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:55.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.602+0000 7f8f3973d700 1 -- 172.21.15.196:0/1391114923 >> 172.21.15.196:0/1391114923 conn(0x7f8f340fab80 msgr2=0x7f8f340fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:55.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.602+0000 7f8f3973d700 1 -- 172.21.15.196:0/1391114923 shutdown_connections 2024-03-26T22:00:55.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.603+0000 7f8f3973d700 1 -- 172.21.15.196:0/1391114923 wait complete. 2024-03-26T22:00:55.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.603+0000 7f8f3973d700 1 Processor -- start 2024-03-26T22:00:55.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.603+0000 7f8f3973d700 1 -- start start 2024-03-26T22:00:55.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f3973d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f3973d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f341960b0 con 0x7f8f34101270 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f32ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f32ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36038/0 (socket says 172.21.15.196:36038) 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f32ffd700 1 -- 172.21.15.196:0/150542702 learned_addr learned my addr 172.21.15.196:0/150542702 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f32ffd700 1 -- 172.21.15.196:0/150542702 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f240095f0 con 0x7f8f34101270 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.604+0000 7f8f32ffd700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34195b70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009ed0 tx=0x7f8f24004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f2401d070 con 0x7f8f34101270 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f341962b0 con 0x7f8f34101270 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f34196750 con 0x7f8f34101270 2024-03-26T22:00:55.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f2400be20 con 0x7f8f34101270 2024-03-26T22:00:55.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f2400f460 con 0x7f8f34101270 2024-03-26T22:00:55.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3418f560 con 0x7f8f34101270 2024-03-26T22:00:55.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.605+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8f240227e0 con 0x7f8f34101270 2024-03-26T22:00:55.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.606+0000 7f8f1bfff700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8f1c0387d0 0x7f8f1c03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:55.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.606+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8f2404c7e0 con 0x7f8f34101270 2024-03-26T22:00:55.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.606+0000 7f8f327fc700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8f1c0387d0 0x7f8f1c03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:55.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.606+0000 7f8f327fc700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8f1c0387d0 0x7f8f1c03ac90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f28006fd0 tx=0x7f8f28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:55.609 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.611+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8f24022470 con 0x7f8f34101270 2024-03-26T22:00:55.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.884+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f340623e0 con 0x7f8f34101270 2024-03-26T22:00:55.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.884+0000 7f8f1bfff700 1 -- 172.21.15.196:0/150542702 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8f340623e0 con 0x7f8f34101270 2024-03-26T22:00:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:55.883 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8f1c0387d0 msgr2=0x7f8f1c03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8f1c0387d0 0x7f8f1c03ac90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f28006fd0 tx=0x7f8f28006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 msgr2=0x7f8f34195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34195b70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009ed0 tx=0x7f8f24004690 comp rx=0 tx=0).stop 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 shutdown_connections 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8f1c0387d0 0x7f8f1c03ac90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 --2- 172.21.15.196:0/150542702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f34101270 0x7f8f34195b70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.886+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 >> 172.21.15.196:0/150542702 conn(0x7f8f340fab80 msgr2=0x7f8f340fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.887+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 shutdown_connections 2024-03-26T22:00:55.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:55.887+0000 7f8f3973d700 1 -- 172.21.15.196:0/150542702 wait complete. 2024-03-26T22:00:55.886 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:00:56.413 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:56 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/150542702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:00:57.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:57 smithi194 ceph-mon[29693]: pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:00:57.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:00:57.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:00:57.739 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:57.785 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 -- 172.21.15.196:0/4214256787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 msgr2=0x7f53200ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 --2- 172.21.15.196:0/4214256787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200ffa40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5314009910 tx=0x7f5314009c20 comp rx=0 tx=0).stop 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 -- 172.21.15.196:0/4214256787 shutdown_connections 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 --2- 172.21.15.196:0/4214256787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200ffa40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 -- 172.21.15.196:0/4214256787 >> 172.21.15.196:0/4214256787 conn(0x7f53200fab80 msgr2=0x7f53200fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 -- 172.21.15.196:0/4214256787 shutdown_connections 2024-03-26T22:00:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.387+0000 7f53263bc700 1 -- 172.21.15.196:0/4214256787 wait complete. 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.388+0000 7f53263bc700 1 Processor -- start 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.388+0000 7f53263bc700 1 -- start start 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.388+0000 7f53263bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.388+0000 7f53263bc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200fcf70 con 0x7f53200ff620 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36052/0 (socket says 172.21.15.196:36052) 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531ffff700 1 -- 172.21.15.196:0/2810836925 learned_addr learned my addr 172.21.15.196:0/2810836925 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531ffff700 1 -- 172.21.15.196:0/2810836925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53140095f0 con 0x7f53200ff620 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531ffff700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200fca30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5314006010 tx=0x7f5314009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f531401d070 con 0x7f53200ff620 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53200fb1b0 con 0x7f53200ff620 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53200fb650 con 0x7f53200ff620 2024-03-26T22:00:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5314005dc0 con 0x7f53200ff620 2024-03-26T22:00:59.390 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.389+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5314022e70 con 0x7f53200ff620 2024-03-26T22:00:59.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.390+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5320105d80 con 0x7f53200ff620 2024-03-26T22:00:59.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.390+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f5314022730 con 0x7f53200ff620 2024-03-26T22:00:59.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.391+0000 7f531d7fa700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5308038770 0x7f530803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:00:59.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.391+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f531404c850 con 0x7f53200ff620 2024-03-26T22:00:59.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.391+0000 7f531f7fe700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5308038770 0x7f530803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:00:59.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.391+0000 7f531f7fe700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5308038770 0x7f530803ac30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5310006fd0 tx=0x7f5310006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:00:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.396+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f531400b960 con 0x7f53200ff620 2024-03-26T22:00:59.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.666+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53200623e0 con 0x7f53200ff620 2024-03-26T22:00:59.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.667+0000 7f531d7fa700 1 -- 172.21.15.196:0/2810836925 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5314027030 con 0x7f53200ff620 2024-03-26T22:00:59.666 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:00:59.666 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:00:59.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.668+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5308038770 msgr2=0x7f530803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:59.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5308038770 0x7f530803ac30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5310006fd0 tx=0x7f5310006e40 comp rx=0 tx=0).stop 2024-03-26T22:00:59.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 msgr2=0x7f53200fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200fca30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5314006010 tx=0x7f5314009f80 comp rx=0 tx=0).stop 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 shutdown_connections 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5308038770 0x7f530803ac30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 --2- 172.21.15.196:0/2810836925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53200ff620 0x7f53200fca30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 >> 172.21.15.196:0/2810836925 conn(0x7f53200fab80 msgr2=0x7f53200682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 shutdown_connections 2024-03-26T22:00:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:00:59.669+0000 7f53263bc700 1 -- 172.21.15.196:0/2810836925 wait complete. 2024-03-26T22:00:59.671 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:00.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:59 smithi194 ceph-mon[29693]: pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:00.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:00:59 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2810836925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:01.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:01.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:01.525 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:01.571 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:02.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:01 smithi194 ceph-mon[29693]: pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:03.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 -- 172.21.15.196:0/244218675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 msgr2=0x7f3ad40ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:03.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 --2- 172.21.15.196:0/244218675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad40ffa30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009910 tx=0x7f3ac4009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:03.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 -- 172.21.15.196:0/244218675 shutdown_connections 2024-03-26T22:01:03.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 --2- 172.21.15.196:0/244218675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad40ffa30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:03.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 -- 172.21.15.196:0/244218675 >> 172.21.15.196:0/244218675 conn(0x7f3ad40fabb0 msgr2=0x7f3ad40fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:03.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 -- 172.21.15.196:0/244218675 shutdown_connections 2024-03-26T22:01:03.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.189+0000 7f3ad923e700 1 -- 172.21.15.196:0/244218675 wait complete. 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad923e700 1 Processor -- start 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad923e700 1 -- start start 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad923e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad4195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad923e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4196140 con 0x7f3ad40ff610 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad2d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad4195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad2d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad4195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:59316/0 (socket says 172.21.15.196:59316) 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.190+0000 7f3ad2d9d700 1 -- 172.21.15.196:0/2213936667 learned_addr learned my addr 172.21.15.196:0/2213936667 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:03.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3ad2d9d700 1 -- 172.21.15.196:0/2213936667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40095f0 con 0x7f3ad40ff610 2024-03-26T22:01:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3ad2d9d700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad4195c00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4006010 tx=0x7f3ac4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ac401d070 con 0x7f3ad40ff610 2024-03-26T22:01:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad4196340 con 0x7f3ad40ff610 2024-03-26T22:01:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ac4005dc0 con 0x7f3ad40ff610 2024-03-26T22:01:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ac4022e90 con 0x7f3ad40ff610 2024-03-26T22:01:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.191+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad41967e0 con 0x7f3ad40ff610 2024-03-26T22:01:03.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.192+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3ac400f550 con 0x7f3ad40ff610 2024-03-26T22:01:03.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.192+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad418f700 con 0x7f3ad40ff610 2024-03-26T22:01:03.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.192+0000 7f3abbfff700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3abc038760 0x7f3abc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:03.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.192+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ac404c7f0 con 0x7f3ad40ff610 2024-03-26T22:01:03.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.193+0000 7f3ad259c700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3abc038760 0x7f3abc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:03.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.193+0000 7f3ad259c700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3abc038760 0x7f3abc03ac20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8006fd0 tx=0x7f3ac8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:03.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.198+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3ac402a950 con 0x7f3ad40ff610 2024-03-26T22:01:03.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.482+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ad40623e0 con 0x7f3ad40ff610 2024-03-26T22:01:03.481 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.483+0000 7f3abbfff700 1 -- 172.21.15.196:0/2213936667 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3ad40623e0 con 0x7f3ad40ff610 2024-03-26T22:01:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:03.481 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:03.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3abc038760 msgr2=0x7f3abc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:03.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3abc038760 0x7f3abc03ac20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8006fd0 tx=0x7f3ac8006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:03.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 msgr2=0x7f3ad4195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:03.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad4195c00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4006010 tx=0x7f3ac4009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 shutdown_connections 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3abc038760 0x7f3abc03ac20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 --2- 172.21.15.196:0/2213936667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ad40ff610 0x7f3ad4195c00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 >> 172.21.15.196:0/2213936667 conn(0x7f3ad40fabb0 msgr2=0x7f3ad4068bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 shutdown_connections 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:03.485+0000 7f3ad923e700 1 -- 172.21.15.196:0/2213936667 wait complete. 2024-03-26T22:01:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:04.163 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:03 smithi194 ceph-mon[29693]: pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:04.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:03 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2213936667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:05.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:05.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:05.315 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:05.352 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:05 smithi194 ceph-mon[29693]: pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:05.360 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 -- 172.21.15.196:0/4171894964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 msgr2=0x7fc5600ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 --2- 172.21.15.196:0/4171894964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc5600ffa40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009910 tx=0x7fc55c009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 -- 172.21.15.196:0/4171894964 shutdown_connections 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 --2- 172.21.15.196:0/4171894964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc5600ffa40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 -- 172.21.15.196:0/4171894964 >> 172.21.15.196:0/4171894964 conn(0x7fc5600fabc0 msgr2=0x7fc5600fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 -- 172.21.15.196:0/4171894964 shutdown_connections 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 -- 172.21.15.196:0/4171894964 wait complete. 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 Processor -- start 2024-03-26T22:01:06.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.965+0000 7fc567a55700 1 -- start start 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc567a55700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc560194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc567a55700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560195010 con 0x7fc5600ff620 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5657f1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc560194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5657f1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc560194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:59322/0 (socket says 172.21.15.196:59322) 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5657f1700 1 -- 172.21.15.196:0/221140819 learned_addr learned my addr 172.21.15.196:0/221140819 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5657f1700 1 -- 172.21.15.196:0/221140819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0095f0 con 0x7fc5600ff620 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5657f1700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc560194ad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc55c006010 tx=0x7fc55c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55c01d070 con 0x7fc5600ff620 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc560195210 con 0x7fc5600ff620 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc55c005dc0 con 0x7fc5600ff620 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55c022e90 con 0x7fc5600ff620 2024-03-26T22:01:06.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.966+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5601956b0 con 0x7fc5600ff620 2024-03-26T22:01:06.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.967+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fc55c00b960 con 0x7fc5600ff620 2024-03-26T22:01:06.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.967+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56018e450 con 0x7fc5600ff620 2024-03-26T22:01:06.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.967+0000 7fc5567fc700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc54c038760 0x7fc54c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:06.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.967+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc55c04c8c0 con 0x7fc5600ff620 2024-03-26T22:01:06.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.968+0000 7fc564ff0700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc54c038760 0x7fc54c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:06.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.968+0000 7fc564ff0700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc54c038760 0x7fc54c03ac20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc550006fd0 tx=0x7fc550006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:06.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:06.973+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc55c022750 con 0x7fc5600ff620 2024-03-26T22:01:07.241 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.243+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5600623e0 con 0x7fc5600ff620 2024-03-26T22:01:07.242 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.243+0000 7fc5567fc700 1 -- 172.21.15.196:0/221140819 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc5600623e0 con 0x7fc5600ff620 2024-03-26T22:01:07.242 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:07.242 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc54c038760 msgr2=0x7fc54c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc54c038760 0x7fc54c03ac20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc550006fd0 tx=0x7fc550006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 msgr2=0x7fc560194ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc560194ad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc55c006010 tx=0x7fc55c009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 shutdown_connections 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc54c038760 0x7fc54c03ac20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 --2- 172.21.15.196:0/221140819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5600ff620 0x7fc560194ad0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 >> 172.21.15.196:0/221140819 conn(0x7fc5600fabc0 msgr2=0x7fc5600fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:07.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 shutdown_connections 2024-03-26T22:01:07.245 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:07.245+0000 7fc567a55700 1 -- 172.21.15.196:0/221140819 wait complete. 2024-03-26T22:01:07.245 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:08.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:07 smithi194 ceph-mon[29693]: pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:08.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:07 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/221140819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:08.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:08.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:09.149 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:09.196 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:10.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:09 smithi194 ceph-mon[29693]: pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:10.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 -- 172.21.15.196:0/637195090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 msgr2=0x7fab50100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:10.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 --2- 172.21.15.196:0/637195090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50100b30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fab40009910 tx=0x7fab40009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:10.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 -- 172.21.15.196:0/637195090 shutdown_connections 2024-03-26T22:01:10.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 --2- 172.21.15.196:0/637195090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50100b30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:10.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 -- 172.21.15.196:0/637195090 >> 172.21.15.196:0/637195090 conn(0x7fab500fbcd0 msgr2=0x7fab500fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:10.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 -- 172.21.15.196:0/637195090 shutdown_connections 2024-03-26T22:01:10.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 -- 172.21.15.196:0/637195090 wait complete. 2024-03-26T22:01:10.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 Processor -- start 2024-03-26T22:01:10.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.696+0000 7fab5674f700 1 -- start start 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab5674f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab5674f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab50074f60 con 0x7fab50100710 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab4ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab4ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:59332/0 (socket says 172.21.15.196:59332) 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab4ffff700 1 -- 172.21.15.196:0/1127295500 learned_addr learned my addr 172.21.15.196:0/1127295500 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab4ffff700 1 -- 172.21.15.196:0/1127295500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab400095f0 con 0x7fab50100710 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.697+0000 7fab4ffff700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50074a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fab40006010 tx=0x7fab40009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.698+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab4001d070 con 0x7fab50100710 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.698+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab40005dc0 con 0x7fab50100710 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.698+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab500731a0 con 0x7fab50100710 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.698+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab40022e90 con 0x7fab50100710 2024-03-26T22:01:10.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.698+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab50073640 con 0x7fab50100710 2024-03-26T22:01:10.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.698+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab50106e70 con 0x7fab50100710 2024-03-26T22:01:10.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.699+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fab40022470 con 0x7fab50100710 2024-03-26T22:01:10.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.699+0000 7fab4d7fa700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fab38038760 0x7fab3803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:10.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.699+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab4004c7f0 con 0x7fab50100710 2024-03-26T22:01:10.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.699+0000 7fab4f7fe700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fab38038760 0x7fab3803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:10.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.699+0000 7fab4f7fe700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fab38038760 0x7fab3803ac20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fab44006fd0 tx=0x7fab44006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:10.702 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.704+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fab4000b960 con 0x7fab50100710 2024-03-26T22:01:10.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.981+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab500623e0 con 0x7fab50100710 2024-03-26T22:01:10.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.981+0000 7fab4d7fa700 1 -- 172.21.15.196:0/1127295500 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fab40027030 con 0x7fab50100710 2024-03-26T22:01:10.981 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:10.981 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:10.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fab38038760 msgr2=0x7fab3803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:10.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fab38038760 0x7fab3803ac20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fab44006fd0 tx=0x7fab44006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:10.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 msgr2=0x7fab50074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50074a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fab40006010 tx=0x7fab40009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 shutdown_connections 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fab38038760 0x7fab3803ac20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 --2- 172.21.15.196:0/1127295500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab50100710 0x7fab50074a20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 >> 172.21.15.196:0/1127295500 conn(0x7fab500fbcd0 msgr2=0x7fab500fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 shutdown_connections 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:10.984+0000 7fab5674f700 1 -- 172.21.15.196:0/1127295500 wait complete. 2024-03-26T22:01:10.983 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:12.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:11 smithi194 ceph-mon[29693]: pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:12.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:11 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1127295500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:12.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:12.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:12.837 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:12.882 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:14.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:13 smithi194 ceph-mon[29693]: pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 -- 172.21.15.196:0/380997320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 msgr2=0x7f2540101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 --2- 172.21.15.196:0/380997320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540101060 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2530009910 tx=0x7f2530009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 -- 172.21.15.196:0/380997320 shutdown_connections 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 --2- 172.21.15.196:0/380997320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540101060 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 -- 172.21.15.196:0/380997320 >> 172.21.15.196:0/380997320 conn(0x7f25400fa310 msgr2=0x7f25400fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 -- 172.21.15.196:0/380997320 shutdown_connections 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.371+0000 7f2548316700 1 -- 172.21.15.196:0/380997320 wait complete. 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f2548316700 1 Processor -- start 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f2548316700 1 -- start start 2024-03-26T22:01:14.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f2548316700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540193a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f2548316700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2540193f70 con 0x7f2540100c40 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f25460b2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540193a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f25460b2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540193a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44710/0 (socket says 172.21.15.196:44710) 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f25460b2700 1 -- 172.21.15.196:0/430382022 learned_addr learned my addr 172.21.15.196:0/430382022 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.372+0000 7f25460b2700 1 -- 172.21.15.196:0/430382022 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300095f0 con 0x7f2540100c40 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.373+0000 7f25460b2700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540193a30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2530006010 tx=0x7f2530009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.373+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f253001d070 con 0x7f2540100c40 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.373+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2540194170 con 0x7f2540100c40 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.373+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2530005dc0 con 0x7f2540100c40 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.373+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2530022e90 con 0x7f2540100c40 2024-03-26T22:01:14.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.373+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2540194610 con 0x7f2540100c40 2024-03-26T22:01:14.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.374+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254018d360 con 0x7f2540100c40 2024-03-26T22:01:14.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.374+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2530022470 con 0x7f2540100c40 2024-03-26T22:01:14.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.374+0000 7f25377fe700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f252c03aa00 0x7f252c03cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:14.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.374+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f253004c760 con 0x7f2540100c40 2024-03-26T22:01:14.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.374+0000 7f25458b1700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f252c03aa00 0x7f252c03cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:14.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.375+0000 7f25458b1700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f252c03aa00 0x7f252c03cec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f253c006fd0 tx=0x7f253c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:14.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.380+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f253000b960 con 0x7f2540100c40 2024-03-26T22:01:14.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.646+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25400623e0 con 0x7f2540100c40 2024-03-26T22:01:14.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.646+0000 7f25377fe700 1 -- 172.21.15.196:0/430382022 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2530027020 con 0x7f2540100c40 2024-03-26T22:01:14.645 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:14.645 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:14.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.648+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f252c03aa00 msgr2=0x7f252c03cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:14.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.648+0000 7f2548316700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f252c03aa00 0x7f252c03cec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f253c006fd0 tx=0x7f253c006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:14.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.648+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 msgr2=0x7f2540193a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:14.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.648+0000 7f2548316700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540193a30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2530006010 tx=0x7f2530009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:14.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.649+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 shutdown_connections 2024-03-26T22:01:14.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.649+0000 7f2548316700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f252c03aa00 0x7f252c03cec0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:14.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.649+0000 7f2548316700 1 --2- 172.21.15.196:0/430382022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2540100c40 0x7f2540193a30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:14.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.649+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 >> 172.21.15.196:0/430382022 conn(0x7f25400fa310 msgr2=0x7f25400fae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:14.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.649+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 shutdown_connections 2024-03-26T22:01:14.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:14.649+0000 7f2548316700 1 -- 172.21.15.196:0/430382022 wait complete. 2024-03-26T22:01:14.648 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:15.366 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:15 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/430382022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:15.366 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:15 smithi194 ceph-mon[29693]: pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:16.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:16.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:16.508 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:16.554 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 -- 172.21.15.196:0/2533037993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 msgr2=0x7f8824103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 --2- 172.21.15.196:0/2533037993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824103630 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8814009910 tx=0x7f8814009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 -- 172.21.15.196:0/2533037993 shutdown_connections 2024-03-26T22:01:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 --2- 172.21.15.196:0/2533037993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824103630 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 -- 172.21.15.196:0/2533037993 >> 172.21.15.196:0/2533037993 conn(0x7f88240fabd0 msgr2=0x7f88240fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 -- 172.21.15.196:0/2533037993 shutdown_connections 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 -- 172.21.15.196:0/2533037993 wait complete. 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.145+0000 7f8829a1b700 1 Processor -- start 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8829a1b700 1 -- start start 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8829a1b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8829a1b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8824196140 con 0x7f8824101240 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8822ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8822ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44730/0 (socket says 172.21.15.196:44730) 2024-03-26T22:01:18.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8822ffd700 1 -- 172.21.15.196:0/131651432 learned_addr learned my addr 172.21.15.196:0/131651432 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8822ffd700 1 -- 172.21.15.196:0/131651432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140095f0 con 0x7f8824101240 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8822ffd700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824195c00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8814005b40 tx=0x7f8814005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f881401d070 con 0x7f8824101240 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8824196340 con 0x7f8824101240 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8814005dc0 con 0x7f8824101240 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.146+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8814022e00 con 0x7f8824101240 2024-03-26T22:01:18.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.147+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88241967e0 con 0x7f8824101240 2024-03-26T22:01:18.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.147+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f881400b960 con 0x7f8824101240 2024-03-26T22:01:18.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.147+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882418f560 con 0x7f8824101240 2024-03-26T22:01:18.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.148+0000 7f8828a19700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f880c038760 0x7f880c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:18.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.148+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f881404c870 con 0x7f8824101240 2024-03-26T22:01:18.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.148+0000 7f88227fc700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f880c038760 0x7f880c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:18.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.148+0000 7f88227fc700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f880c038760 0x7f880c03ac20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8818006fd0 tx=0x7f8818006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:18.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.153+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f881400bdf0 con 0x7f8824101240 2024-03-26T22:01:18.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:17 smithi194 ceph-mon[29693]: pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:18.423 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.425+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88240623e0 con 0x7f8824101240 2024-03-26T22:01:18.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.425+0000 7f8828a19700 1 -- 172.21.15.196:0/131651432 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f88240623e0 con 0x7f8824101240 2024-03-26T22:01:18.424 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:18.424 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:18.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f880c038760 msgr2=0x7f880c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:18.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f880c038760 0x7f880c03ac20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8818006fd0 tx=0x7f8818006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:18.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 msgr2=0x7f8824195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:18.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824195c00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8814005b40 tx=0x7f8814005000 comp rx=0 tx=0).stop 2024-03-26T22:01:18.426 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 shutdown_connections 2024-03-26T22:01:18.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f880c038760 0x7f880c03ac20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:18.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 --2- 172.21.15.196:0/131651432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8824101240 0x7f8824195c00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:18.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 >> 172.21.15.196:0/131651432 conn(0x7f88240fabd0 msgr2=0x7f88240fb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:18.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 shutdown_connections 2024-03-26T22:01:18.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:18.427+0000 7f8829a1b700 1 -- 172.21.15.196:0/131651432 wait complete. 2024-03-26T22:01:18.427 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:19.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:18 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/131651432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:19.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:19.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:20.153 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:20.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:19 smithi194 ceph-mon[29693]: pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:20.200 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:21.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.854+0000 7f3310461700 1 -- 172.21.15.196:0/333029743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 msgr2=0x7f33080fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:21.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.854+0000 7f3310461700 1 --2- 172.21.15.196:0/333029743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f33080fee10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3300009910 tx=0x7f3300009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:21.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 -- 172.21.15.196:0/333029743 shutdown_connections 2024-03-26T22:01:21.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 --2- 172.21.15.196:0/333029743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f33080fee10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 -- 172.21.15.196:0/333029743 >> 172.21.15.196:0/333029743 conn(0x7f33080fa330 msgr2=0x7f33080fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 -- 172.21.15.196:0/333029743 shutdown_connections 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 -- 172.21.15.196:0/333029743 wait complete. 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 Processor -- start 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 -- start start 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f3308101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.855+0000 7f3310461700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3308101740 con 0x7f33080fe9f0 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f330e1fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f3308101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f330e1fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f3308101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44748/0 (socket says 172.21.15.196:44748) 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f330e1fd700 1 -- 172.21.15.196:0/4135976805 learned_addr learned my addr 172.21.15.196:0/4135976805 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f330e1fd700 1 -- 172.21.15.196:0/4135976805 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33000095f0 con 0x7f33080fe9f0 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f330e1fd700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f3308101200 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3300006010 tx=0x7f3300009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f330001d070 con 0x7f33080fe9f0 2024-03-26T22:01:21.856 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33080ff980 con 0x7f33080fe9f0 2024-03-26T22:01:21.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3300005dc0 con 0x7f33080fe9f0 2024-03-26T22:01:21.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3300022e90 con 0x7f33080fe9f0 2024-03-26T22:01:21.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.856+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33080ffe20 con 0x7f33080fe9f0 2024-03-26T22:01:21.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.857+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f330000f4f0 con 0x7f33080fe9f0 2024-03-26T22:01:21.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.857+0000 7f33077fe700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f32f0038760 0x7f32f003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:21.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.857+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3308106ea0 con 0x7f33080fe9f0 2024-03-26T22:01:21.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.857+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f330004c6e0 con 0x7f33080fe9f0 2024-03-26T22:01:21.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.857+0000 7f330d9fc700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f32f0038760 0x7f32f003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:21.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.858+0000 7f330d9fc700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f32f0038760 0x7f32f003ac20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00ad30 tx=0x7f32fc0093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:21.861 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:21.863+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3300027070 con 0x7f33080fe9f0 2024-03-26T22:01:22.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.134+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33080623e0 con 0x7f33080fe9f0 2024-03-26T22:01:22.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.135+0000 7f33077fe700 1 -- 172.21.15.196:0/4135976805 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3300034070 con 0x7f33080fe9f0 2024-03-26T22:01:22.133 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:22.133 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f32f0038760 msgr2=0x7f32f003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f32f0038760 0x7f32f003ac20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00ad30 tx=0x7f32fc0093f0 comp rx=0 tx=0).stop 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 msgr2=0x7f3308101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f3308101200 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3300006010 tx=0x7f3300009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 shutdown_connections 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f32f0038760 0x7f32f003ac20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 --2- 172.21.15.196:0/4135976805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33080fe9f0 0x7f3308101200 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 >> 172.21.15.196:0/4135976805 conn(0x7f33080fa330 msgr2=0x7f33080fae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 shutdown_connections 2024-03-26T22:01:22.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:22.137+0000 7f3310461700 1 -- 172.21.15.196:0/4135976805 wait complete. 2024-03-26T22:01:22.137 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:22.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:21 smithi194 ceph-mon[29693]: pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:23.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:22 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4135976805' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:23.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:23.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:23.878 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:23.922 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:24.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:23 smithi194 ceph-mon[29693]: pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:25.379 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:25 smithi194 ceph-mon[29693]: pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:25.517 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/4174174125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 msgr2=0x7fd6ac100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:25.517 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 --2- 172.21.15.196:0/4174174125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac100b30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009910 tx=0x7fd69c009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:25.517 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/4174174125 shutdown_connections 2024-03-26T22:01:25.517 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 --2- 172.21.15.196:0/4174174125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac100b30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/4174174125 >> 172.21.15.196:0/4174174125 conn(0x7fd6ac0fbcd0 msgr2=0x7fd6ac0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/4174174125 shutdown_connections 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.507+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/4174174125 wait complete. 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b2e0b700 1 Processor -- start 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b2e0b700 1 -- start start 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b2e0b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b2e0b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac1960f0 con 0x7fd6ac100710 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b0ba7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b0ba7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac195bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:41802/0 (socket says 172.21.15.196:41802) 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b0ba7700 1 -- 172.21.15.196:0/1280704728 learned_addr learned my addr 172.21.15.196:0/1280704728 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:25.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b0ba7700 1 -- 172.21.15.196:0/1280704728 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0095f0 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.508+0000 7fd6b0ba7700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac195bb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006010 tx=0x7fd69c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.509+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69c01d070 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.509+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac1962f0 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.509+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd69c005dc0 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.509+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69c022e90 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.509+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac196790 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.510+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac18f540 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.510+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fd69c022470 con 0x7fd6ac100710 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.510+0000 7fd6a9ffb700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd694038760 0x7fd69403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:25.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.510+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd69c04c7f0 con 0x7fd6ac100710 2024-03-26T22:01:25.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.510+0000 7fd6abfff700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd694038760 0x7fd69403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:25.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.511+0000 7fd6abfff700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd694038760 0x7fd69403ac20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0006fd0 tx=0x7fd6a0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:25.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.516+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd69c00b960 con 0x7fd6ac100710 2024-03-26T22:01:25.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.784+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6ac0623e0 con 0x7fd6ac100710 2024-03-26T22:01:25.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.785+0000 7fd6a9ffb700 1 -- 172.21.15.196:0/1280704728 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd6ac0623e0 con 0x7fd6ac100710 2024-03-26T22:01:25.784 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:25.784 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:25.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd694038760 msgr2=0x7fd69403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:25.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd694038760 0x7fd69403ac20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0006fd0 tx=0x7fd6a0006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:25.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 msgr2=0x7fd6ac195bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:25.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac195bb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006010 tx=0x7fd69c009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:25.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 shutdown_connections 2024-03-26T22:01:25.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd694038760 0x7fd69403ac20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:25.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 --2- 172.21.15.196:0/1280704728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6ac100710 0x7fd6ac195bb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:25.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 >> 172.21.15.196:0/1280704728 conn(0x7fd6ac0fbcd0 msgr2=0x7fd6ac0fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:25.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.787+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 shutdown_connections 2024-03-26T22:01:25.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:25.788+0000 7fd6b2e0b700 1 -- 172.21.15.196:0/1280704728 wait complete. 2024-03-26T22:01:25.787 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:26.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:26 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1280704728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:27.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:27.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:27.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:27 smithi194 ceph-mon[29693]: pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:27.592 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:27.638 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:29.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/1058297333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 msgr2=0x7f9dd8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 --2- 172.21.15.196:0/1058297333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8100b80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009910 tx=0x7f9dcc009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/1058297333 shutdown_connections 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 --2- 172.21.15.196:0/1058297333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8100b80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/1058297333 >> 172.21.15.196:0/1058297333 conn(0x7f9dd80fbd40 msgr2=0x7f9dd80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/1058297333 shutdown_connections 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.055+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/1058297333 wait complete. 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.056+0000 7f9ddd2ea700 1 Processor -- start 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.056+0000 7f9ddd2ea700 1 -- start start 2024-03-26T22:01:29.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.056+0000 7f9ddd2ea700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:29.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.056+0000 7f9ddd2ea700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8198330 con 0x7f9dd8100760 2024-03-26T22:01:29.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dd6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:29.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dd6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:41818/0 (socket says 172.21.15.196:41818) 2024-03-26T22:01:29.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dd6ffd700 1 -- 172.21.15.196:0/3224624435 learned_addr learned my addr 172.21.15.196:0/3224624435 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:29.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dd6ffd700 1 -- 172.21.15.196:0/3224624435 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc0095f0 con 0x7f9dd8100760 2024-03-26T22:01:29.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dd6ffd700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8197df0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc006010 tx=0x7f9dcc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:29.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dcc01d070 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd8198530 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dcc005dc0 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dcc022e90 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.057+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd81989d0 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.058+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9dcc00f4f0 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.058+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd8191990 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.058+0000 7f9dbffff700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dc00387b0 0x7f9dc003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.059+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9dcc04cab0 con 0x7f9dd8100760 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.059+0000 7f9dd67fc700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dc00387b0 0x7f9dc003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.059+0000 7f9dd67fc700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dc00387b0 0x7f9dc003ac70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8006fd0 tx=0x7f9dc8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:29.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.064+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9dcc02a950 con 0x7f9dd8100760 2024-03-26T22:01:29.332 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.334+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dd80623e0 con 0x7f9dd8100760 2024-03-26T22:01:29.333 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.334+0000 7f9dbffff700 1 -- 172.21.15.196:0/3224624435 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9dd80623e0 con 0x7f9dd8100760 2024-03-26T22:01:29.333 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:29.333 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dc00387b0 msgr2=0x7f9dc003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dc00387b0 0x7f9dc003ac70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8006fd0 tx=0x7f9dc8006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 msgr2=0x7f9dd8197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8197df0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc006010 tx=0x7f9dcc009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 shutdown_connections 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9dc00387b0 0x7f9dc003ac70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:29.335 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 --2- 172.21.15.196:0/3224624435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9dd8100760 0x7f9dd8197df0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.336+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 >> 172.21.15.196:0/3224624435 conn(0x7f9dd80fbd40 msgr2=0x7f9dd80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.337+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 shutdown_connections 2024-03-26T22:01:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:29.337+0000 7f9ddd2ea700 1 -- 172.21.15.196:0/3224624435 wait complete. 2024-03-26T22:01:29.336 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:30.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:29 smithi194 ceph-mon[29693]: pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:30.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:29 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3224624435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:30.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:30.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:31.184 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:31.230 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:32.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:31 smithi194 ceph-mon[29693]: pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 -- 172.21.15.196:0/784012784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 msgr2=0x7fa5dc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 --2- 172.21.15.196:0/784012784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc0ffa30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa5d400b0a0 tx=0x7fa5d400b3b0 comp rx=0 tx=0).stop 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 -- 172.21.15.196:0/784012784 shutdown_connections 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 --2- 172.21.15.196:0/784012784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc0ffa30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 -- 172.21.15.196:0/784012784 >> 172.21.15.196:0/784012784 conn(0x7fa5dc0fabb0 msgr2=0x7fa5dc0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 -- 172.21.15.196:0/784012784 shutdown_connections 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.823+0000 7fa5e3128700 1 -- 172.21.15.196:0/784012784 wait complete. 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e3128700 1 Processor -- start 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e3128700 1 -- start start 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e3128700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc069160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e3128700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc0696a0 con 0x7fa5dc0ff610 2024-03-26T22:01:32.824 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e0ec4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc069160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e0ec4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc069160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42268/0 (socket says 172.21.15.196:42268) 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.824+0000 7fa5e0ec4700 1 -- 172.21.15.196:0/3084808303 learned_addr learned my addr 172.21.15.196:0/3084808303 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5e0ec4700 1 -- 172.21.15.196:0/3084808303 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4009d00 con 0x7fa5dc0ff610 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5e0ec4700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc069160 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40062a0 tx=0x7fa5d400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5d400e040 con 0x7fa5dc0ff610 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5d4003d70 con 0x7fa5dc0ff610 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5dc198900 con 0x7fa5dc0ff610 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5d401c890 con 0x7fa5dc0ff610 2024-03-26T22:01:32.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.825+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5dc198d40 con 0x7fa5dc0ff610 2024-03-26T22:01:32.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.826+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5dc18e440 con 0x7fa5dc0ff610 2024-03-26T22:01:32.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.826+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa5d401cb50 con 0x7fa5dc0ff610 2024-03-26T22:01:32.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.826+0000 7fa5d9ffb700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa5c80387c0 0x7fa5c803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:32.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.826+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa5d404ca60 con 0x7fa5dc0ff610 2024-03-26T22:01:32.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.826+0000 7fa5dbfff700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa5c80387c0 0x7fa5c803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:32.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.827+0000 7fa5dbfff700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa5c80387c0 0x7fa5c803ac80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4006fd0 tx=0x7fa5c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:32.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:32.831+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa5d401a3b0 con 0x7fa5dc0ff610 2024-03-26T22:01:33.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.099+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5dc0623e0 con 0x7fa5dc0ff610 2024-03-26T22:01:33.098 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.100+0000 7fa5d9ffb700 1 -- 172.21.15.196:0/3084808303 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa5d401a3b0 con 0x7fa5dc0ff610 2024-03-26T22:01:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:33.099 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa5c80387c0 msgr2=0x7fa5c803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa5c80387c0 0x7fa5c803ac80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4006fd0 tx=0x7fa5c4006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 msgr2=0x7fa5dc069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc069160 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40062a0 tx=0x7fa5d400bd40 comp rx=0 tx=0).stop 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 shutdown_connections 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa5c80387c0 0x7fa5c803ac80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 --2- 172.21.15.196:0/3084808303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5dc0ff610 0x7fa5dc069160 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 >> 172.21.15.196:0/3084808303 conn(0x7fa5dc0fabb0 msgr2=0x7fa5dc0fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 shutdown_connections 2024-03-26T22:01:33.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:33.102+0000 7fa5e3128700 1 -- 172.21.15.196:0/3084808303 wait complete. 2024-03-26T22:01:33.102 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:33 smithi194 ceph-mon[29693]: pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:33 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3084808303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:34.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:34.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:34.942 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:34.988 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:35.392 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:35 smithi194 ceph-mon[29693]: pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:36.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 -- 172.21.15.196:0/1376215882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 msgr2=0x7f5ea40ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 --2- 172.21.15.196:0/1376215882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea40ffa40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009910 tx=0x7f5e9c009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 -- 172.21.15.196:0/1376215882 shutdown_connections 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 --2- 172.21.15.196:0/1376215882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea40ffa40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 -- 172.21.15.196:0/1376215882 >> 172.21.15.196:0/1376215882 conn(0x7f5ea40fab80 msgr2=0x7f5ea40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 -- 172.21.15.196:0/1376215882 shutdown_connections 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.440+0000 7f5eab7e0700 1 -- 172.21.15.196:0/1376215882 wait complete. 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5eab7e0700 1 Processor -- start 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5eab7e0700 1 -- start start 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5eab7e0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea4194ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5eab7e0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4195000 con 0x7f5ea40ff620 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5ea957c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea4194ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:36.441 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5ea957c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea4194ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42284/0 (socket says 172.21.15.196:42284) 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.441+0000 7f5ea957c700 1 -- 172.21.15.196:0/2899226020 learned_addr learned my addr 172.21.15.196:0/2899226020 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5ea957c700 1 -- 172.21.15.196:0/2899226020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0095f0 con 0x7f5ea40ff620 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5ea957c700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea4194ac0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c006010 tx=0x7f5e9c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9c01f070 con 0x7f5ea40ff620 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea4195200 con 0x7f5ea40ff620 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e9c005dc0 con 0x7f5ea40ff620 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea41956a0 con 0x7f5ea40ff620 2024-03-26T22:01:36.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.442+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9c0245c0 con 0x7f5ea40ff620 2024-03-26T22:01:36.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.443+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea40623e0 con 0x7f5ea40ff620 2024-03-26T22:01:36.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.443+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f5e9c019650 con 0x7f5ea40ff620 2024-03-26T22:01:36.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.443+0000 7f5e9a7fc700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5e900387c0 0x7f5e9003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:36.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.443+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5e9c04e9a0 con 0x7f5ea40ff620 2024-03-26T22:01:36.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.444+0000 7f5ea8d7b700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5e900387c0 0x7f5e9003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:36.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.444+0000 7f5ea8d7b700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5e900387c0 0x7f5e9003ac80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5e94006fd0 tx=0x7f5e94006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:36.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.449+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e9c00bce0 con 0x7f5ea40ff620 2024-03-26T22:01:36.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.723+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ea4105650 con 0x7f5ea40ff620 2024-03-26T22:01:36.722 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.724+0000 7f5e9a7fc700 1 -- 172.21.15.196:0/2899226020 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5e9c029070 con 0x7f5ea40ff620 2024-03-26T22:01:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:36.723 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5e900387c0 msgr2=0x7f5e9003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5e900387c0 0x7f5e9003ac80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5e94006fd0 tx=0x7f5e94006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 msgr2=0x7f5ea4194ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea4194ac0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c006010 tx=0x7f5e9c009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 shutdown_connections 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5e900387c0 0x7f5e9003ac80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 --2- 172.21.15.196:0/2899226020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5ea40ff620 0x7f5ea4194ac0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 >> 172.21.15.196:0/2899226020 conn(0x7f5ea40fab80 msgr2=0x7f5ea40fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 shutdown_connections 2024-03-26T22:01:36.725 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:36.726+0000 7f5eab7e0700 1 -- 172.21.15.196:0/2899226020 wait complete. 2024-03-26T22:01:36.726 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:38.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:37 smithi194 ceph-mon[29693]: pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:38.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:37 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2899226020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:38.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:38.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:38.591 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:38.637 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:40.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:39 smithi194 ceph-mon[29693]: pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 -- 172.21.15.196:0/2858831710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 msgr2=0x7f06ec0ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 --2- 172.21.15.196:0/2858831710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec0ff150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c210 tx=0x7f06e800c520 comp rx=0 tx=0).stop 2024-03-26T22:01:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 -- 172.21.15.196:0/2858831710 shutdown_connections 2024-03-26T22:01:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 --2- 172.21.15.196:0/2858831710 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec0ff150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 -- 172.21.15.196:0/2858831710 >> 172.21.15.196:0/2858831710 conn(0x7f06ec0fa290 msgr2=0x7f06ec0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 -- 172.21.15.196:0/2858831710 shutdown_connections 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.250+0000 7f06f2ee2700 1 -- 172.21.15.196:0/2858831710 wait complete. 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f2ee2700 1 Processor -- start 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f2ee2700 1 -- start start 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f2ee2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f2ee2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec196130 con 0x7f06ec0fed30 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f0c7e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f0c7e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42310/0 (socket says 172.21.15.196:42310) 2024-03-26T22:01:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.251+0000 7f06f0c7e700 1 -- 172.21.15.196:0/3968468 learned_addr learned my addr 172.21.15.196:0/3968468 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06f0c7e700 1 -- 172.21.15.196:0/3968468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e800ae10 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06f0c7e700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec195bf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f06e8016040 tx=0x7f06e8008460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e800f040 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06e800a2e0 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec196330 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e8007c10 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.252+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec196830 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.253+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec18f710 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.253+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f06e8007630 con 0x7f06ec0fed30 2024-03-26T22:01:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.254+0000 7f06e1ffb700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f06d40387c0 0x7f06d403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:40.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.254+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f06e804c3e0 con 0x7f06ec0fed30 2024-03-26T22:01:40.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.254+0000 7f06e3fff700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f06d40387c0 0x7f06d403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:40.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.254+0000 7f06e3fff700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f06d40387c0 0x7f06d403ac80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06dc006fd0 tx=0x7f06dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:40.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.259+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f06e8026540 con 0x7f06ec0fed30 2024-03-26T22:01:40.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.526+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06ec0623e0 con 0x7f06ec0fed30 2024-03-26T22:01:40.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.526+0000 7f06e1ffb700 1 -- 172.21.15.196:0/3968468 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f06e8026360 con 0x7f06ec0fed30 2024-03-26T22:01:40.525 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:40.525 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:40.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f06d40387c0 msgr2=0x7f06d403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f06d40387c0 0x7f06d403ac80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06dc006fd0 tx=0x7f06dc006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 msgr2=0x7f06ec195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec195bf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f06e8016040 tx=0x7f06e8008460 comp rx=0 tx=0).stop 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 shutdown_connections 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f06d40387c0 0x7f06d403ac80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 --2- 172.21.15.196:0/3968468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06ec0fed30 0x7f06ec195bf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 >> 172.21.15.196:0/3968468 conn(0x7f06ec0fa290 msgr2=0x7f06ec18a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 shutdown_connections 2024-03-26T22:01:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:40.529+0000 7f06f2ee2700 1 -- 172.21.15.196:0/3968468 wait complete. 2024-03-26T22:01:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:41.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:40 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3968468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:42.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:42.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:42.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:41 smithi194 ceph-mon[29693]: pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:42.346 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:42.392 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 -- 172.21.15.196:0/3065892744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 msgr2=0x7fa178100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 --2- 172.21.15.196:0/3065892744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178100b30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa174009910 tx=0x7fa174009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 -- 172.21.15.196:0/3065892744 shutdown_connections 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 --2- 172.21.15.196:0/3065892744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178100b30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 -- 172.21.15.196:0/3065892744 >> 172.21.15.196:0/3065892744 conn(0x7fa1780fbcd0 msgr2=0x7fa1780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 -- 172.21.15.196:0/3065892744 shutdown_connections 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.996+0000 7fa17ff9e700 1 -- 172.21.15.196:0/3065892744 wait complete. 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17ff9e700 1 Processor -- start 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17ff9e700 1 -- start start 2024-03-26T22:01:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17ff9e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:43.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17ff9e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa178196120 con 0x7fa178100710 2024-03-26T22:01:43.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17dd3a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:43.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17dd3a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:48050/0 (socket says 172.21.15.196:48050) 2024-03-26T22:01:43.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17dd3a700 1 -- 172.21.15.196:0/240308994 learned_addr learned my addr 172.21.15.196:0/240308994 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:43.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.997+0000 7fa17dd3a700 1 -- 172.21.15.196:0/240308994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740095f0 con 0x7fa178100710 2024-03-26T22:01:43.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.998+0000 7fa17dd3a700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178195be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa174006010 tx=0x7fa174009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:43.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.998+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa17401d070 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.998+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa178196320 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.998+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa174005dc0 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.998+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa174022e90 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.998+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1781967c0 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.999+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17818f540 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.999+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa174022470 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.999+0000 7fa16effd700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa164038760 0x7fa16403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.999+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa17404c8b0 con 0x7fa178100710 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:43.999+0000 7fa17d539700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa164038760 0x7fa16403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.000+0000 7fa17d539700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa164038760 0x7fa16403ac20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa168006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:44.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.004+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa17400f4f0 con 0x7fa178100710 2024-03-26T22:01:44.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:43 smithi194 ceph-mon[29693]: pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:44.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.277+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1780623e0 con 0x7fa178100710 2024-03-26T22:01:44.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.278+0000 7fa16effd700 1 -- 172.21.15.196:0/240308994 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa174027070 con 0x7fa178100710 2024-03-26T22:01:44.277 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:44.277 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa164038760 msgr2=0x7fa16403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa164038760 0x7fa16403ac20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa168006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 msgr2=0x7fa178195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178195be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa174006010 tx=0x7fa174009f80 comp rx=0 tx=0).stop 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 shutdown_connections 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa164038760 0x7fa16403ac20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 --2- 172.21.15.196:0/240308994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa178100710 0x7fa178195be0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 >> 172.21.15.196:0/240308994 conn(0x7fa1780fbcd0 msgr2=0x7fa1780fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 shutdown_connections 2024-03-26T22:01:44.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:44.280+0000 7fa17ff9e700 1 -- 172.21.15.196:0/240308994 wait complete. 2024-03-26T22:01:44.280 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:45.405 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:45 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/240308994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:45.405 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:45 smithi194 ceph-mon[29693]: pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:45.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:45.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:46.015 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:46.061 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:47.624 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 -- 172.21.15.196:0/3546809249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 msgr2=0x7fef10100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 --2- 172.21.15.196:0/3546809249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10100b40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fef0800b0a0 tx=0x7fef0800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 -- 172.21.15.196:0/3546809249 shutdown_connections 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 --2- 172.21.15.196:0/3546809249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10100b40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 -- 172.21.15.196:0/3546809249 >> 172.21.15.196:0/3546809249 conn(0x7fef100fbcc0 msgr2=0x7fef100fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 -- 172.21.15.196:0/3546809249 shutdown_connections 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.625+0000 7fef187ee700 1 -- 172.21.15.196:0/3546809249 wait complete. 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef187ee700 1 Processor -- start 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef187ee700 1 -- start start 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef187ee700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:47.625 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef187ee700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef101960d0 con 0x7fef10100720 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef1658a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef1658a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:48066/0 (socket says 172.21.15.196:48066) 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.626+0000 7fef1658a700 1 -- 172.21.15.196:0/2288641144 learned_addr learned my addr 172.21.15.196:0/2288641144 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef1658a700 1 -- 172.21.15.196:0/2288641144 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef08009d00 con 0x7fef10100720 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef1658a700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10195b90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fef080062a0 tx=0x7fef0800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef0800e040 con 0x7fef10100720 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef101962d0 con 0x7fef10100720 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef08003d70 con 0x7fef10100720 2024-03-26T22:01:47.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef0801c890 con 0x7fef10100720 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.627+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef10196770 con 0x7fef10100720 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.628+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1018f550 con 0x7fef10100720 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.628+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fef0801cb50 con 0x7fef10100720 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.628+0000 7fef077fe700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feefc0387c0 0x7feefc03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.628+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fef0804ca60 con 0x7fef10100720 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.628+0000 7fef15d89700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feefc0387c0 0x7feefc03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:47.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.629+0000 7fef15d89700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feefc0387c0 0x7feefc03ac80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef00006fd0 tx=0x7fef00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:47.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.634+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef0801a3b0 con 0x7fef10100720 2024-03-26T22:01:47.906 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.907+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef100623e0 con 0x7fef10100720 2024-03-26T22:01:47.906 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.908+0000 7fef077fe700 1 -- 172.21.15.196:0/2288641144 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fef100623e0 con 0x7fef10100720 2024-03-26T22:01:47.907 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:47.907 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feefc0387c0 msgr2=0x7feefc03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feefc0387c0 0x7feefc03ac80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef00006fd0 tx=0x7fef00006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 msgr2=0x7fef10195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10195b90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fef080062a0 tx=0x7fef0800bd40 comp rx=0 tx=0).stop 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 shutdown_connections 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feefc0387c0 0x7feefc03ac80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 --2- 172.21.15.196:0/2288641144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fef10100720 0x7fef10195b90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 >> 172.21.15.196:0/2288641144 conn(0x7fef100fbcc0 msgr2=0x7fef100fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 shutdown_connections 2024-03-26T22:01:47.909 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:47.910+0000 7fef187ee700 1 -- 172.21.15.196:0/2288641144 wait complete. 2024-03-26T22:01:47.910 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:48.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:47 smithi194 ceph-mon[29693]: pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:49.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:48 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2288641144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:49.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:49.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:49.762 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:49.807 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:50.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:49 smithi194 ceph-mon[29693]: pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:51.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 -- 172.21.15.196:0/2998432141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 msgr2=0x7feae01035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 --2- 172.21.15.196:0/2998432141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae01035d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fead0009910 tx=0x7fead0009c20 comp rx=0 tx=0).stop 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 -- 172.21.15.196:0/2998432141 shutdown_connections 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 --2- 172.21.15.196:0/2998432141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae01035d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 -- 172.21.15.196:0/2998432141 >> 172.21.15.196:0/2998432141 conn(0x7feae00fabf0 msgr2=0x7feae00fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 -- 172.21.15.196:0/2998432141 shutdown_connections 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.345+0000 7feae4e14700 1 -- 172.21.15.196:0/2998432141 wait complete. 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feae4e14700 1 Processor -- start 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feae4e14700 1 -- start start 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feae4e14700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae0100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feae4e14700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0100f50 con 0x7feae01011e0 2024-03-26T22:01:51.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feade59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae0100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feade59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae0100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:48076/0 (socket says 172.21.15.196:48076) 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feade59c700 1 -- 172.21.15.196:0/2871008042 learned_addr learned my addr 172.21.15.196:0/2871008042 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.346+0000 7feade59c700 1 -- 172.21.15.196:0/2871008042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead00095f0 con 0x7feae01011e0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.347+0000 7feade59c700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae0100a10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fead0009ed0 tx=0x7fead0004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.347+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead001d070 con 0x7feae01011e0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.347+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae00ff190 con 0x7feae01011e0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.347+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fead000be20 con 0x7feae01011e0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.347+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead000f460 con 0x7feae01011e0 2024-03-26T22:01:51.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.347+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae00ff630 con 0x7feae01011e0 2024-03-26T22:01:51.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.348+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae0106eb0 con 0x7feae01011e0 2024-03-26T22:01:51.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.348+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fead0022800 con 0x7feae01011e0 2024-03-26T22:01:51.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.348+0000 7feacf7fe700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feac80387d0 0x7feac803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:51.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.348+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fead004ccc0 con 0x7feae01011e0 2024-03-26T22:01:51.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.348+0000 7feaddd9b700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feac80387d0 0x7feac803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:51.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.349+0000 7feaddd9b700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feac80387d0 0x7feac803ac90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fead4006fd0 tx=0x7fead4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:51.352 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.353+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fead0053080 con 0x7feae01011e0 2024-03-26T22:01:51.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.628+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feae00623e0 con 0x7feae01011e0 2024-03-26T22:01:51.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.629+0000 7feacf7fe700 1 -- 172.21.15.196:0/2871008042 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fead002bba0 con 0x7feae01011e0 2024-03-26T22:01:51.630 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:51.630 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feac80387d0 msgr2=0x7feac803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feac80387d0 0x7feac803ac90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fead4006fd0 tx=0x7fead4006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 msgr2=0x7feae0100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae0100a10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fead0009ed0 tx=0x7fead0004690 comp rx=0 tx=0).stop 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 shutdown_connections 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feac80387d0 0x7feac803ac90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 --2- 172.21.15.196:0/2871008042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae01011e0 0x7feae0100a10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 >> 172.21.15.196:0/2871008042 conn(0x7feae00fabf0 msgr2=0x7feae00fb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:51.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 shutdown_connections 2024-03-26T22:01:51.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:51.631+0000 7feae4e14700 1 -- 172.21.15.196:0/2871008042 wait complete. 2024-03-26T22:01:51.632 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:52.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:51 smithi194 ceph-mon[29693]: pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:52.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:51 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2871008042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:53.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:53.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:53.515 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:53.560 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:54.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:53 smithi194 ceph-mon[29693]: pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2486290447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 msgr2=0x7fc5e8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 --2- 172.21.15.196:0/2486290447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8100b40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000b0a0 tx=0x7fc5e000b3b0 comp rx=0 tx=0).stop 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2486290447 shutdown_connections 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 --2- 172.21.15.196:0/2486290447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8100b40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2486290447 >> 172.21.15.196:0/2486290447 conn(0x7fc5e80fbcc0 msgr2=0x7fc5e80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2486290447 shutdown_connections 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.125+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2486290447 wait complete. 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5ee1e1700 1 Processor -- start 2024-03-26T22:01:55.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5ee1e1700 1 -- start start 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5ee1e1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5ee1e1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81982f0 con 0x7fc5e8100720 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5e77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5e77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45666/0 (socket says 172.21.15.196:45666) 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.126+0000 7fc5e77fe700 1 -- 172.21.15.196:0/2118773798 learned_addr learned my addr 172.21.15.196:0/2118773798 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5e77fe700 1 -- 172.21.15.196:0/2118773798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0009d00 con 0x7fc5e8100720 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5e77fe700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8197db0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc5e00062a0 tx=0x7fc5e000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5e000e040 con 0x7fc5e8100720 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e81984f0 con 0x7fc5e8100720 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e8198990 con 0x7fc5e8100720 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5e0003d70 con 0x7fc5e8100720 2024-03-26T22:01:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.127+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5e001c7b0 con 0x7fc5e8100720 2024-03-26T22:01:55.128 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.128+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8005320 con 0x7fc5e8100720 2024-03-26T22:01:55.128 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.128+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fc5e001ca70 con 0x7fc5e8100720 2024-03-26T22:01:55.128 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.128+0000 7fc5e4ff9700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc5d40387c0 0x7fc5d403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:55.128 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.128+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5e004c910 con 0x7fc5e8100720 2024-03-26T22:01:55.129 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.128+0000 7fc5e6ffd700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc5d40387c0 0x7fc5d403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:55.129 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.129+0000 7fc5e6ffd700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc5d40387c0 0x7fc5d403ac80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0006fd0 tx=0x7fc5d0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:55.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.133+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc5e0024d30 con 0x7fc5e8100720 2024-03-26T22:01:55.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.402+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5c8005190 con 0x7fc5e8100720 2024-03-26T22:01:55.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.403+0000 7fc5e4ff9700 1 -- 172.21.15.196:0/2118773798 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc5e002be80 con 0x7fc5e8100720 2024-03-26T22:01:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:55.402 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:55.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc5d40387c0 msgr2=0x7fc5d403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc5d40387c0 0x7fc5d403ac80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0006fd0 tx=0x7fc5d0006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 msgr2=0x7fc5e8197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8197db0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc5e00062a0 tx=0x7fc5e000bd40 comp rx=0 tx=0).stop 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 shutdown_connections 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc5d40387c0 0x7fc5d403ac80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 --2- 172.21.15.196:0/2118773798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5e8100720 0x7fc5e8197db0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 >> 172.21.15.196:0/2118773798 conn(0x7fc5e80fbcc0 msgr2=0x7fc5e80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 shutdown_connections 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:55.405+0000 7fc5ee1e1700 1 -- 172.21.15.196:0/2118773798 wait complete. 2024-03-26T22:01:55.404 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:01:55.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:55 smithi194 ceph-mon[29693]: pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:56.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:56 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2118773798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:01:57.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:01:57.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:01:57.259 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:57.305 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:01:57.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:57 smithi194 ceph-mon[29693]: pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 -- 172.21.15.196:0/16849755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 msgr2=0x7f0014100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 --2- 172.21.15.196:0/16849755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014100b40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f000c00b0a0 tx=0x7f000c00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 -- 172.21.15.196:0/16849755 shutdown_connections 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 --2- 172.21.15.196:0/16849755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014100b40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 -- 172.21.15.196:0/16849755 >> 172.21.15.196:0/16849755 conn(0x7f00140fbcc0 msgr2=0x7f00140fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 -- 172.21.15.196:0/16849755 shutdown_connections 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.910+0000 7f001bfc0700 1 -- 172.21.15.196:0/16849755 wait complete. 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f001bfc0700 1 Processor -- start 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f001bfc0700 1 -- start start 2024-03-26T22:01:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f001bfc0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f001bfc0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0014196100 con 0x7f0014100720 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f0019d5c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f0019d5c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45672/0 (socket says 172.21.15.196:45672) 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f0019d5c700 1 -- 172.21.15.196:0/1670003041 learned_addr learned my addr 172.21.15.196:0/1670003041 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.911+0000 7f0019d5c700 1 -- 172.21.15.196:0/1670003041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c009d00 con 0x7f0014100720 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.912+0000 7f0019d5c700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014195bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f000c0062a0 tx=0x7f000c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.912+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f000c00e040 con 0x7f0014100720 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.912+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0014196300 con 0x7f0014100720 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.912+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f000c003d70 con 0x7f0014100720 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.912+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f000c01c890 con 0x7f0014100720 2024-03-26T22:01:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.912+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00141967a0 con 0x7f0014100720 2024-03-26T22:01:58.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.913+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f000c01cb50 con 0x7f0014100720 2024-03-26T22:01:58.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.913+0000 7f000affd700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f00000387c0 0x7f000003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:01:58.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.913+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001418f550 con 0x7f0014100720 2024-03-26T22:01:58.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.913+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f000c04cb60 con 0x7f0014100720 2024-03-26T22:01:58.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.913+0000 7f001955b700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f00000387c0 0x7f000003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:01:58.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.913+0000 7f001955b700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f00000387c0 0x7f000003ac80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0004006fd0 tx=0x7f0004006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:01:58.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:58.918+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f000c01b770 con 0x7f0014100720 2024-03-26T22:01:59.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.187+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00140623e0 con 0x7f0014100720 2024-03-26T22:01:59.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.188+0000 7f000affd700 1 -- 172.21.15.196:0/1670003041 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f000c011390 con 0x7f0014100720 2024-03-26T22:01:59.187 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:01:59.187 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:01:59.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f00000387c0 msgr2=0x7f000003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:59.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f00000387c0 0x7f000003ac80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0004006fd0 tx=0x7f0004006e40 comp rx=0 tx=0).stop 2024-03-26T22:01:59.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 msgr2=0x7f0014195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:01:59.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014195bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f000c0062a0 tx=0x7f000c00bd40 comp rx=0 tx=0).stop 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 shutdown_connections 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f00000387c0 0x7f000003ac80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 --2- 172.21.15.196:0/1670003041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0014100720 0x7f0014195bc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 >> 172.21.15.196:0/1670003041 conn(0x7f00140fbcc0 msgr2=0x7f00140fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 shutdown_connections 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:01:59.190+0000 7f001bfc0700 1 -- 172.21.15.196:0/1670003041 wait complete. 2024-03-26T22:01:59.190 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:00.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:59 smithi194 ceph-mon[29693]: pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:00.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:01:59 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1670003041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:00.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:00.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:01.074 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:01.121 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:02.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:01 smithi194 ceph-mon[29693]: pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:02.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 -- 172.21.15.196:0/2285350847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 msgr2=0x7ff8c41033e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:02.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 --2- 172.21.15.196:0/2285350847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41033e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009910 tx=0x7ff8b8009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:02.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 -- 172.21.15.196:0/2285350847 shutdown_connections 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 --2- 172.21.15.196:0/2285350847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41033e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 -- 172.21.15.196:0/2285350847 >> 172.21.15.196:0/2285350847 conn(0x7ff8c40faa00 msgr2=0x7ff8c40fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 -- 172.21.15.196:0/2285350847 shutdown_connections 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 -- 172.21.15.196:0/2285350847 wait complete. 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.526+0000 7ff8c89a0700 1 Processor -- start 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c89a0700 1 -- start start 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c89a0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c89a0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4195f10 con 0x7ff8c4100ff0 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60370/0 (socket says 172.21.15.196:60370) 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c37fe700 1 -- 172.21.15.196:0/3542904182 learned_addr learned my addr 172.21.15.196:0/3542904182 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:02.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c37fe700 1 -- 172.21.15.196:0/3542904182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80095f0 con 0x7ff8c4100ff0 2024-03-26T22:02:02.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.527+0000 7ff8c37fe700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41959d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009ed0 tx=0x7ff8b8004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:02.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.528+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b801d070 con 0x7ff8c4100ff0 2024-03-26T22:02:02.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.528+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c4196110 con 0x7ff8c4100ff0 2024-03-26T22:02:02.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.528+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8b800be20 con 0x7ff8c4100ff0 2024-03-26T22:02:02.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.528+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b800f460 con 0x7ff8c4100ff0 2024-03-26T22:02:02.528 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.528+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c41965b0 con 0x7ff8c4100ff0 2024-03-26T22:02:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.529+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c418f390 con 0x7ff8c4100ff0 2024-03-26T22:02:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.529+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7ff8b800b960 con 0x7ff8c4100ff0 2024-03-26T22:02:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.529+0000 7ff8c0ff9700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff8ac0387d0 0x7ff8ac03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.529+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff8b804c930 con 0x7ff8c4100ff0 2024-03-26T22:02:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.529+0000 7ff8c2ffd700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff8ac0387d0 0x7ff8ac03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.530+0000 7ff8c2ffd700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff8ac0387d0 0x7ff8ac03ac90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4006fd0 tx=0x7ff8b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:02.533 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.534+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff8b802a360 con 0x7ff8c4100ff0 2024-03-26T22:02:02.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.804+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8c40623e0 con 0x7ff8c4100ff0 2024-03-26T22:02:02.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.809+0000 7ff8c0ff9700 1 -- 172.21.15.196:0/3542904182 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff8c40623e0 con 0x7ff8c4100ff0 2024-03-26T22:02:02.808 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:02.808 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:02.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff8ac0387d0 msgr2=0x7ff8ac03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff8ac0387d0 0x7ff8ac03ac90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4006fd0 tx=0x7ff8b4006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 msgr2=0x7ff8c41959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41959d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009ed0 tx=0x7ff8b8004690 comp rx=0 tx=0).stop 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 shutdown_connections 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7ff8ac0387d0 0x7ff8ac03ac90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 --2- 172.21.15.196:0/3542904182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8c4100ff0 0x7ff8c41959d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.811+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 >> 172.21.15.196:0/3542904182 conn(0x7ff8c40faa00 msgr2=0x7ff8c40fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.812+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 shutdown_connections 2024-03-26T22:02:02.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:02.812+0000 7ff8c89a0700 1 -- 172.21.15.196:0/3542904182 wait complete. 2024-03-26T22:02:02.812 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:04.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:03 smithi194 ceph-mon[29693]: pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:04.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:03 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3542904182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:04.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:04.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:04.646 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:04.692 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:05.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:05 smithi194 ceph-mon[29693]: pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 -- 172.21.15.196:0/4105343702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 msgr2=0x7fdd50100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 --2- 172.21.15.196:0/4105343702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50100fe0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009910 tx=0x7fdd40009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 -- 172.21.15.196:0/4105343702 shutdown_connections 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 --2- 172.21.15.196:0/4105343702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50100fe0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 -- 172.21.15.196:0/4105343702 >> 172.21.15.196:0/4105343702 conn(0x7fdd500fa310 msgr2=0x7fdd500fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 -- 172.21.15.196:0/4105343702 shutdown_connections 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.181+0000 7fdd554ca700 1 -- 172.21.15.196:0/4105343702 wait complete. 2024-03-26T22:02:06.181 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd554ca700 1 Processor -- start 2024-03-26T22:02:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd554ca700 1 -- start start 2024-03-26T22:02:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd554ca700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd554ca700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd50196170 con 0x7fdd50100bc0 2024-03-26T22:02:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd4effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd4effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50195c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60396/0 (socket says 172.21.15.196:60396) 2024-03-26T22:02:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd4effd700 1 -- 172.21.15.196:0/771809775 learned_addr learned my addr 172.21.15.196:0/771809775 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.182+0000 7fdd4effd700 1 -- 172.21.15.196:0/771809775 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd400095f0 con 0x7fdd50100bc0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.183+0000 7fdd4effd700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50195c30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdd40006010 tx=0x7fdd40009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.183+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd4001d070 con 0x7fdd50100bc0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.183+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd50196370 con 0x7fdd50100bc0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.183+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd40005dc0 con 0x7fdd50100bc0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.183+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd50196810 con 0x7fdd50100bc0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.183+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd40022e50 con 0x7fdd50100bc0 2024-03-26T22:02:06.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.184+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fdd4000b960 con 0x7fdd50100bc0 2024-03-26T22:02:06.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.184+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5018f730 con 0x7fdd50100bc0 2024-03-26T22:02:06.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.184+0000 7fdd37fff700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdd38038770 0x7fdd3803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:06.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.184+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd4004c870 con 0x7fdd50100bc0 2024-03-26T22:02:06.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.184+0000 7fdd4e7fc700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdd38038770 0x7fdd3803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:06.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.185+0000 7fdd4e7fc700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdd38038770 0x7fdd3803ac30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd44006fd0 tx=0x7fdd44006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:06.188 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.189+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd4002a750 con 0x7fdd50100bc0 2024-03-26T22:02:06.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.460+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd500623e0 con 0x7fdd50100bc0 2024-03-26T22:02:06.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.460+0000 7fdd37fff700 1 -- 172.21.15.196:0/771809775 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fdd500623e0 con 0x7fdd50100bc0 2024-03-26T22:02:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:06.460 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:06.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdd38038770 msgr2=0x7fdd3803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:06.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdd38038770 0x7fdd3803ac30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd44006fd0 tx=0x7fdd44006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:06.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 msgr2=0x7fdd50195c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:06.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50195c30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdd40006010 tx=0x7fdd40009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 shutdown_connections 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdd38038770 0x7fdd3803ac30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 --2- 172.21.15.196:0/771809775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd50100bc0 0x7fdd50195c30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.462+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 >> 172.21.15.196:0/771809775 conn(0x7fdd500fa310 msgr2=0x7fdd500fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.463+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 shutdown_connections 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:06.463+0000 7fdd554ca700 1 -- 172.21.15.196:0/771809775 wait complete. 2024-03-26T22:02:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:06.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:06 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/771809775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:07.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:07 smithi194 ceph-mon[29693]: pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:08.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:08.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:08.331 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:09.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 -- 172.21.15.196:0/3417686168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 msgr2=0x7f3ce0102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:09.912 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 --2- 172.21.15.196:0/3417686168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce0102130 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009910 tx=0x7f3cd0009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 -- 172.21.15.196:0/3417686168 shutdown_connections 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 --2- 172.21.15.196:0/3417686168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce0102130 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 -- 172.21.15.196:0/3417686168 >> 172.21.15.196:0/3417686168 conn(0x7f3ce00fa370 msgr2=0x7f3ce00fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 -- 172.21.15.196:0/3417686168 shutdown_connections 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.913+0000 7f3ce5e79700 1 -- 172.21.15.196:0/3417686168 wait complete. 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3ce5e79700 1 Processor -- start 2024-03-26T22:02:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3ce5e79700 1 -- start start 2024-03-26T22:02:09.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3ce5e79700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce01012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3ce5e79700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce0101820 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3cdf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce01012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3cdf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce01012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60416/0 (socket says 172.21.15.196:60416) 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3cdf7fe700 1 -- 172.21.15.196:0/1764136169 learned_addr learned my addr 172.21.15.196:0/1764136169 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.914+0000 7f3cdf7fe700 1 -- 172.21.15.196:0/1764136169 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00095f0 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.915+0000 7f3cdf7fe700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce01012e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0006010 tx=0x7f3cd0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.915+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cd001d070 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.915+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cd0005dc0 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.915+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce00ffa60 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.915+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cd0022e90 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.915+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce00fff00 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.916+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce0106e90 con 0x7f3ce0101d10 2024-03-26T22:02:09.915 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.916+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3cd0022470 con 0x7f3ce0101d10 2024-03-26T22:02:09.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.916+0000 7f3cdcff9700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3cc8038760 0x7f3cc803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:09.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.916+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3cd004c8b0 con 0x7f3ce0101d10 2024-03-26T22:02:09.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.916+0000 7f3cdeffd700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3cc8038760 0x7f3cc803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:09.916 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.917+0000 7f3cdeffd700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3cc8038760 0x7f3cc803ac20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4006fd0 tx=0x7f3cd4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:09.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:09.921+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3cd000b960 con 0x7f3ce0101d10 2024-03-26T22:02:10.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:09 smithi194 ceph-mon[29693]: pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:10.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.191+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ce00623e0 con 0x7f3ce0101d10 2024-03-26T22:02:10.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.191+0000 7f3cdcff9700 1 -- 172.21.15.196:0/1764136169 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3cd0027030 con 0x7f3ce0101d10 2024-03-26T22:02:10.191 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:10.191 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:10.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3cc8038760 msgr2=0x7f3cc803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:10.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3cc8038760 0x7f3cc803ac20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4006fd0 tx=0x7f3cd4006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:10.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 msgr2=0x7f3ce01012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce01012e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0006010 tx=0x7f3cd0009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 shutdown_connections 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3cc8038760 0x7f3cc803ac20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 --2- 172.21.15.196:0/1764136169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ce0101d10 0x7f3ce01012e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 >> 172.21.15.196:0/1764136169 conn(0x7f3ce00fa370 msgr2=0x7f3ce00fae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 shutdown_connections 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:10.193+0000 7f3ce5e79700 1 -- 172.21.15.196:0/1764136169 wait complete. 2024-03-26T22:02:10.193 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:11.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:10 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1764136169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:11.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:11.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:12.046 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:12.092 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:12.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:11 smithi194 ceph-mon[29693]: pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:13.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 -- 172.21.15.196:0/1671007434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 msgr2=0x7fa1681035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 --2- 172.21.15.196:0/1671007434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681035b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa158009910 tx=0x7fa158009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 -- 172.21.15.196:0/1671007434 shutdown_connections 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 --2- 172.21.15.196:0/1671007434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681035b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 -- 172.21.15.196:0/1671007434 >> 172.21.15.196:0/1671007434 conn(0x7fa1680fabd0 msgr2=0x7fa1680fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 -- 172.21.15.196:0/1671007434 shutdown_connections 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.641+0000 7fa16e007700 1 -- 172.21.15.196:0/1671007434 wait complete. 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa16e007700 1 Processor -- start 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa16e007700 1 -- start start 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa16e007700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:13.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa16e007700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa168100f30 con 0x7fa1681011c0 2024-03-26T22:02:13.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa1677fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:13.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa1677fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:51362/0 (socket says 172.21.15.196:51362) 2024-03-26T22:02:13.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa1677fe700 1 -- 172.21.15.196:0/2628060195 learned_addr learned my addr 172.21.15.196:0/2628060195 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:13.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.642+0000 7fa1677fe700 1 -- 172.21.15.196:0/2628060195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580095f0 con 0x7fa1681011c0 2024-03-26T22:02:13.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.643+0000 7fa1677fe700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681009f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa158006010 tx=0x7fa1580050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.643+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15801d070 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.643+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1680ff170 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.643+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa15800be20 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.643+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15800f460 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.643+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1680ff610 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.644+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa168106e90 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.644+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa158022470 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.644+0000 7fa164ff9700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa1500387d0 0x7fa15003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.644+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa15804cc30 con 0x7fa1681011c0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.644+0000 7fa166ffd700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa1500387d0 0x7fa15003ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:13.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.645+0000 7fa166ffd700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa1500387d0 0x7fa15003ac90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa15c006fd0 tx=0x7fa15c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:13.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.649+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa158005c00 con 0x7fa1681011c0 2024-03-26T22:02:13.919 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.920+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1680623e0 con 0x7fa1681011c0 2024-03-26T22:02:13.920 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.921+0000 7fa164ff9700 1 -- 172.21.15.196:0/2628060195 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa15802bba0 con 0x7fa1681011c0 2024-03-26T22:02:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:13.920 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:13.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa1500387d0 msgr2=0x7fa15003ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:13.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa1500387d0 0x7fa15003ac90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa15c006fd0 tx=0x7fa15c006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:13.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 msgr2=0x7fa1681009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:13.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681009f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa158006010 tx=0x7fa1580050b0 comp rx=0 tx=0).stop 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 shutdown_connections 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa1500387d0 0x7fa15003ac90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 --2- 172.21.15.196:0/2628060195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1681011c0 0x7fa1681009f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 >> 172.21.15.196:0/2628060195 conn(0x7fa1680fabd0 msgr2=0x7fa1680fb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 shutdown_connections 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:13.923+0000 7fa16e007700 1 -- 172.21.15.196:0/2628060195 wait complete. 2024-03-26T22:02:13.923 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:14.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:13 smithi194 ceph-mon[29693]: pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:15.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:14 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2628060195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:15.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:15.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:15.780 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:15.827 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:16.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:15 smithi194 ceph-mon[29693]: pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:17.350 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/588764390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 msgr2=0x7f8ce01035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 --2- 172.21.15.196:0/588764390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce01035b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8cd800b0a0 tx=0x7f8cd800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/588764390 shutdown_connections 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 --2- 172.21.15.196:0/588764390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce01035b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/588764390 >> 172.21.15.196:0/588764390 conn(0x7f8ce00fabd0 msgr2=0x7f8ce00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/588764390 shutdown_connections 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/588764390 wait complete. 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.351+0000 7f8ce5d2c700 1 Processor -- start 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8ce5d2c700 1 -- start start 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8ce5d2c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce0195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8ce5d2c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0196140 con 0x7f8ce01011c0 2024-03-26T22:02:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8cdf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce0195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8cdf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce0195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:51380/0 (socket says 172.21.15.196:51380) 2024-03-26T22:02:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8cdf7fe700 1 -- 172.21.15.196:0/1675129620 learned_addr learned my addr 172.21.15.196:0/1675129620 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.352+0000 7f8cdf7fe700 1 -- 172.21.15.196:0/1675129620 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8009d00 con 0x7f8ce01011c0 2024-03-26T22:02:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.353+0000 7f8cdf7fe700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce0195c00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80062a0 tx=0x7f8cd800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.353+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd800e040 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.353+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce0196340 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.353+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8cd8003cd0 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.353+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd8012450 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.353+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce01967e0 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.354+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce018f560 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.354+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8cd8004040 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.354+0000 7f8cdcff9700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8cc80387b0 0x7f8cc803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.354+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8cd804c600 con 0x7f8ce01011c0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.354+0000 7f8cdeffd700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8cc80387b0 0x7f8cc803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:17.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.355+0000 7f8cdeffd700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8cc80387b0 0x7f8cc803ac70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4006fd0 tx=0x7f8cd4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:17.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.359+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8cd8024cf0 con 0x7f8ce01011c0 2024-03-26T22:02:17.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.632+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ce00623e0 con 0x7f8ce01011c0 2024-03-26T22:02:17.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.632+0000 7f8cdcff9700 1 -- 172.21.15.196:0/1675129620 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8ce00623e0 con 0x7f8ce01011c0 2024-03-26T22:02:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:17.631 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:17.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8cc80387b0 msgr2=0x7f8cc803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:17.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8cc80387b0 0x7f8cc803ac70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4006fd0 tx=0x7f8cd4006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:17.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 msgr2=0x7f8ce0195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:17.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce0195c00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80062a0 tx=0x7f8cd800bd40 comp rx=0 tx=0).stop 2024-03-26T22:02:17.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 shutdown_connections 2024-03-26T22:02:17.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8cc80387b0 0x7f8cc803ac70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:17.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 --2- 172.21.15.196:0/1675129620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ce01011c0 0x7f8ce0195c00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:17.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 >> 172.21.15.196:0/1675129620 conn(0x7f8ce00fabd0 msgr2=0x7f8ce00fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:17.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 shutdown_connections 2024-03-26T22:02:17.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:17.634+0000 7f8ce5d2c700 1 -- 172.21.15.196:0/1675129620 wait complete. 2024-03-26T22:02:17.634 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:18.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:17 smithi194 ceph-mon[29693]: pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:18.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:17 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1675129620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:19.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:19.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:19.503 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:19.547 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:20.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:19 smithi194 ceph-mon[29693]: pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:21.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/1871494006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 msgr2=0x7fb1cc0fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:21.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 --2- 172.21.15.196:0/1871494006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc0fee50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009910 tx=0x7fb1bc009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:21.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/1871494006 shutdown_connections 2024-03-26T22:02:21.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 --2- 172.21.15.196:0/1871494006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc0fee50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:21.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/1871494006 >> 172.21.15.196:0/1871494006 conn(0x7fb1cc0fa370 msgr2=0x7fb1cc0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:21.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/1871494006 shutdown_connections 2024-03-26T22:02:21.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.046+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/1871494006 wait complete. 2024-03-26T22:02:21.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.047+0000 7fb1d1f1a700 1 Processor -- start 2024-03-26T22:02:21.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.047+0000 7fb1d1f1a700 1 -- start start 2024-03-26T22:02:21.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.047+0000 7fb1d1f1a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:21.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.047+0000 7fb1d1f1a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc1960b0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1cb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1cb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:51396/0 (socket says 172.21.15.196:51396) 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1cb7fe700 1 -- 172.21.15.196:0/2411382514 learned_addr learned my addr 172.21.15.196:0/2411382514 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1cb7fe700 1 -- 172.21.15.196:0/2411382514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc0095f0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1cb7fe700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc195b70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc006010 tx=0x7fb1bc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1bc01d070 con 0x7fb1cc0fea30 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1bc005dc0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc196310 con 0x7fb1cc0fea30 2024-03-26T22:02:21.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1bc022e90 con 0x7fb1cc0fea30 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.048+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc198fe0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.049+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb1bc022470 con 0x7fb1cc0fea30 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.049+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc18f520 con 0x7fb1cc0fea30 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.049+0000 7fb1c8ff9700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1b4038760 0x7fb1b403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.049+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb1bc04c7d0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.050+0000 7fb1caffd700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1b4038760 0x7fb1b403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:21.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.050+0000 7fb1caffd700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1b4038760 0x7fb1b403ac20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:21.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.055+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1bc00b960 con 0x7fb1cc0fea30 2024-03-26T22:02:21.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.324+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1cc0623e0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.325+0000 7fb1c8ff9700 1 -- 172.21.15.196:0/2411382514 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb1cc0623e0 con 0x7fb1cc0fea30 2024-03-26T22:02:21.324 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:21.324 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1b4038760 msgr2=0x7fb1b403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1b4038760 0x7fb1b403ac20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 msgr2=0x7fb1cc195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc195b70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc006010 tx=0x7fb1bc009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 shutdown_connections 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1b4038760 0x7fb1b403ac20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 --2- 172.21.15.196:0/2411382514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1cc0fea30 0x7fb1cc195b70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 >> 172.21.15.196:0/2411382514 conn(0x7fb1cc0fa370 msgr2=0x7fb1cc0fae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:21.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 shutdown_connections 2024-03-26T22:02:21.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:21.327+0000 7fb1d1f1a700 1 -- 172.21.15.196:0/2411382514 wait complete. 2024-03-26T22:02:21.327 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:22.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:21 smithi194 ceph-mon[29693]: pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:22.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:21 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2411382514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:22.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:22.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:23.189 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:23.235 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:24.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:23 smithi194 ceph-mon[29693]: pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 -- 172.21.15.196:0/4098108436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 msgr2=0x7fd9540ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 --2- 172.21.15.196:0/4098108436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd9540ffa30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00b0a0 tx=0x7fd94c00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 -- 172.21.15.196:0/4098108436 shutdown_connections 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 --2- 172.21.15.196:0/4098108436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd9540ffa30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 -- 172.21.15.196:0/4098108436 >> 172.21.15.196:0/4098108436 conn(0x7fd9540fabb0 msgr2=0x7fd9540fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 -- 172.21.15.196:0/4098108436 shutdown_connections 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.933+0000 7fd95c58f700 1 -- 172.21.15.196:0/4098108436 wait complete. 2024-03-26T22:02:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95c58f700 1 Processor -- start 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95c58f700 1 -- start start 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95c58f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd954069540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95c58f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd954069a80 con 0x7fd9540ff610 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95a32b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd954069540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95a32b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd954069540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34274/0 (socket says 172.21.15.196:34274) 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95a32b700 1 -- 172.21.15.196:0/2328470466 learned_addr learned my addr 172.21.15.196:0/2328470466 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95a32b700 1 -- 172.21.15.196:0/2328470466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c009d00 con 0x7fd9540ff610 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.934+0000 7fd95a32b700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd954069540 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0062a0 tx=0x7fd94c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.935+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd94c00e040 con 0x7fd9540ff610 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.935+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd954198930 con 0x7fd9540ff610 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.935+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd94c003d70 con 0x7fd9540ff610 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.935+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd94c01c890 con 0x7fd9540ff610 2024-03-26T22:02:24.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.935+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd954198d70 con 0x7fd9540ff610 2024-03-26T22:02:24.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.935+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95418e440 con 0x7fd9540ff610 2024-03-26T22:02:24.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.936+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fd94c01cb50 con 0x7fd9540ff610 2024-03-26T22:02:24.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.936+0000 7fd94b7fe700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9400387c0 0x7fd94003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:24.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.936+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd94c04ca60 con 0x7fd9540ff610 2024-03-26T22:02:24.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.936+0000 7fd959b2a700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9400387c0 0x7fd94003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:24.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.937+0000 7fd959b2a700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9400387c0 0x7fd94003ac80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd944006fd0 tx=0x7fd944006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:24.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:24.941+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd94c01a3b0 con 0x7fd9540ff610 2024-03-26T22:02:25.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.212+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9540623e0 con 0x7fd9540ff610 2024-03-26T22:02:25.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.213+0000 7fd94b7fe700 1 -- 172.21.15.196:0/2328470466 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd94c02be80 con 0x7fd9540ff610 2024-03-26T22:02:25.212 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:25.212 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:25.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9400387c0 msgr2=0x7fd94003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:25.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9400387c0 0x7fd94003ac80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd944006fd0 tx=0x7fd944006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:25.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 msgr2=0x7fd954069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:25.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd954069540 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0062a0 tx=0x7fd94c00bd40 comp rx=0 tx=0).stop 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 shutdown_connections 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fd9400387c0 0x7fd94003ac80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 --2- 172.21.15.196:0/2328470466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9540ff610 0x7fd954069540 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 >> 172.21.15.196:0/2328470466 conn(0x7fd9540fabb0 msgr2=0x7fd9540fb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 shutdown_connections 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:25.215+0000 7fd95c58f700 1 -- 172.21.15.196:0/2328470466 wait complete. 2024-03-26T22:02:25.215 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:25.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:25 smithi194 ceph-mon[29693]: pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:26.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:26 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2328470466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:26.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:26.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:27.042 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:27.087 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:27.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:27 smithi194 ceph-mon[29693]: pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:28.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 -- 172.21.15.196:0/3299703841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 msgr2=0x7fc69c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:28.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 --2- 172.21.15.196:0/3299703841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0ffa40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009910 tx=0x7fc68c009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 -- 172.21.15.196:0/3299703841 shutdown_connections 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 --2- 172.21.15.196:0/3299703841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0ffa40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 -- 172.21.15.196:0/3299703841 >> 172.21.15.196:0/3299703841 conn(0x7fc69c0fab80 msgr2=0x7fc69c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 -- 172.21.15.196:0/3299703841 shutdown_connections 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.790+0000 7fc6a43b3700 1 -- 172.21.15.196:0/3299703841 wait complete. 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a43b3700 1 Processor -- start 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a43b3700 1 -- start start 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a43b3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a43b3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c0fcf70 con 0x7fc69c0ff620 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a214f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a214f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34284/0 (socket says 172.21.15.196:34284) 2024-03-26T22:02:28.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.791+0000 7fc6a214f700 1 -- 172.21.15.196:0/2694419454 learned_addr learned my addr 172.21.15.196:0/2694419454 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:28.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6a214f700 1 -- 172.21.15.196:0/2694419454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c0095f0 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6a214f700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0fca30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc68c006010 tx=0x7fc68c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c01d070 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c0fb1b0 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc68c005dc0 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c0fb650 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.792+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c022e50 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.793+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fc68c00f460 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.793+0000 7fc6937fe700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc688038770 0x7fc68803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.793+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c105d80 con 0x7fc69c0ff620 2024-03-26T22:02:28.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.793+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc68c04cac0 con 0x7fc69c0ff620 2024-03-26T22:02:28.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.793+0000 7fc6a194e700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc688038770 0x7fc68803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:28.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.794+0000 7fc6a194e700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc688038770 0x7fc68803ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc698006fd0 tx=0x7fc698006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:28.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:28.798+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc68c027030 con 0x7fc69c0ff620 2024-03-26T22:02:29.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.071+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc69c0623e0 con 0x7fc69c0ff620 2024-03-26T22:02:29.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.072+0000 7fc6937fe700 1 -- 172.21.15.196:0/2694419454 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc68c017440 con 0x7fc69c0ff620 2024-03-26T22:02:29.071 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:29.071 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc688038770 msgr2=0x7fc68803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc688038770 0x7fc68803ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc698006fd0 tx=0x7fc698006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 msgr2=0x7fc69c0fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0fca30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc68c006010 tx=0x7fc68c009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 shutdown_connections 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fc688038770 0x7fc68803ac30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 --2- 172.21.15.196:0/2694419454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc69c0ff620 0x7fc69c0fca30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 >> 172.21.15.196:0/2694419454 conn(0x7fc69c0fab80 msgr2=0x7fc69c0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 shutdown_connections 2024-03-26T22:02:29.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:29.074+0000 7fc6a43b3700 1 -- 172.21.15.196:0/2694419454 wait complete. 2024-03-26T22:02:29.074 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:30.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:29 smithi194 ceph-mon[29693]: pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:30.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:29 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2694419454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:30.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:30.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:31.078 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:31.123 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:32.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:31 smithi194 ceph-mon[29693]: pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:32.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.714+0000 7faebff51700 1 -- 172.21.15.196:0/1180662890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 msgr2=0x7faeb8100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:32.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.714+0000 7faebff51700 1 --2- 172.21.15.196:0/1180662890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8100b50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7faeb000b0a0 tx=0x7faeb000b3b0 comp rx=0 tx=0).stop 2024-03-26T22:02:32.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.714+0000 7faebff51700 1 -- 172.21.15.196:0/1180662890 shutdown_connections 2024-03-26T22:02:32.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.714+0000 7faebff51700 1 --2- 172.21.15.196:0/1180662890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8100b50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:32.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.714+0000 7faebff51700 1 -- 172.21.15.196:0/1180662890 >> 172.21.15.196:0/1180662890 conn(0x7faeb80fbcd0 msgr2=0x7faeb80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.715+0000 7faebff51700 1 -- 172.21.15.196:0/1180662890 shutdown_connections 2024-03-26T22:02:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.715+0000 7faebff51700 1 -- 172.21.15.196:0/1180662890 wait complete. 2024-03-26T22:02:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.715+0000 7faebff51700 1 Processor -- start 2024-03-26T22:02:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.715+0000 7faebff51700 1 -- start start 2024-03-26T22:02:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.715+0000 7faebff51700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.715+0000 7faebff51700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb81960f0 con 0x7faeb8100730 2024-03-26T22:02:32.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebdced700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:32.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebdced700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8195bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44244/0 (socket says 172.21.15.196:44244) 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebdced700 1 -- 172.21.15.196:0/772939984 learned_addr learned my addr 172.21.15.196:0/772939984 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebdced700 1 -- 172.21.15.196:0/772939984 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb0009d00 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebdced700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8195bb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faeb000bc90 tx=0x7faeb000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faeb000e040 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb81962f0 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faeb0003d50 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faeb001c8f0 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.716+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb8196790 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.717+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb818f560 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.717+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7faeb001cbb0 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.717+0000 7faeb6ffd700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7faea0038810 0x7faea003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.718+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faeb004ca00 con 0x7faeb8100730 2024-03-26T22:02:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.718+0000 7faebd4ec700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7faea0038810 0x7faea003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:32.718 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.718+0000 7faebd4ec700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7faea0038810 0x7faea003acd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faeac006fd0 tx=0x7faeac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:32.721 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.723+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faeb001c400 con 0x7faeb8100730 2024-03-26T22:02:32.991 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.992+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faeb80623e0 con 0x7faeb8100730 2024-03-26T22:02:32.991 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.993+0000 7faeb6ffd700 1 -- 172.21.15.196:0/772939984 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7faeb80623e0 con 0x7faeb8100730 2024-03-26T22:02:32.992 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:32.992 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7faea0038810 msgr2=0x7faea003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7faea0038810 0x7faea003acd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faeac006fd0 tx=0x7faeac006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 msgr2=0x7faeb8195bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8195bb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faeb000bc90 tx=0x7faeb000bd70 comp rx=0 tx=0).stop 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 shutdown_connections 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7faea0038810 0x7faea003acd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 --2- 172.21.15.196:0/772939984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faeb8100730 0x7faeb8195bb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 >> 172.21.15.196:0/772939984 conn(0x7faeb80fbcd0 msgr2=0x7faeb80fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 shutdown_connections 2024-03-26T22:02:32.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:32.995+0000 7faebff51700 1 -- 172.21.15.196:0/772939984 wait complete. 2024-03-26T22:02:32.995 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:33 smithi194 ceph-mon[29693]: pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:33 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/772939984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:34.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:34.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:34.857 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:34.903 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:35.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:35 smithi194 ceph-mon[29693]: pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:36.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 -- 172.21.15.196:0/2828932943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 msgr2=0x7f3ee41035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:36.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 --2- 172.21.15.196:0/2828932943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee41035a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00b0a0 tx=0x7f3edc00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:02:36.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 -- 172.21.15.196:0/2828932943 shutdown_connections 2024-03-26T22:02:36.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 --2- 172.21.15.196:0/2828932943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee41035a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:36.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 -- 172.21.15.196:0/2828932943 >> 172.21.15.196:0/2828932943 conn(0x7f3ee40fab80 msgr2=0x7f3ee40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:36.508 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 -- 172.21.15.196:0/2828932943 shutdown_connections 2024-03-26T22:02:36.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.508+0000 7f3ee93c1700 1 -- 172.21.15.196:0/2828932943 wait complete. 2024-03-26T22:02:36.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.509+0000 7f3ee93c1700 1 Processor -- start 2024-03-26T22:02:36.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.509+0000 7f3ee93c1700 1 -- start start 2024-03-26T22:02:36.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.509+0000 7f3ee93c1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee4195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:36.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.509+0000 7f3ee93c1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee41960a0 con 0x7f3ee41011b0 2024-03-26T22:02:36.510 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee4195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee4195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44264/0 (socket says 172.21.15.196:44264) 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee2ffd700 1 -- 172.21.15.196:0/612428112 learned_addr learned my addr 172.21.15.196:0/612428112 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee2ffd700 1 -- 172.21.15.196:0/612428112 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc009d00 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee2ffd700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee4195b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0062a0 tx=0x7f3edc00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3edc00e040 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee41962a0 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3edc003d70 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3edc01c8d0 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.510+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee4196740 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.511+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee418f550 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.511+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3edc012640 con 0x7f3ee41011b0 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.511+0000 7f3ecbfff700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3ecc0387c0 0x7f3ecc03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:36.511 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.511+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3edc04ca50 con 0x7f3ee41011b0 2024-03-26T22:02:36.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.512+0000 7f3ee27fc700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3ecc0387c0 0x7f3ecc03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:36.512 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.512+0000 7f3ee27fc700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3ecc0387c0 0x7f3ecc03ac80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8006fd0 tx=0x7f3ed8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:36.515 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.516+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3edc01a590 con 0x7f3ee41011b0 2024-03-26T22:02:36.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.785+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ee40623e0 con 0x7f3ee41011b0 2024-03-26T22:02:36.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.785+0000 7f3ecbfff700 1 -- 172.21.15.196:0/612428112 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3ee40623e0 con 0x7f3ee41011b0 2024-03-26T22:02:36.785 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:36.785 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.787+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3ecc0387c0 msgr2=0x7f3ecc03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3ecc0387c0 0x7f3ecc03ac80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8006fd0 tx=0x7f3ed8006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 msgr2=0x7f3ee4195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee4195b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0062a0 tx=0x7f3edc00bd40 comp rx=0 tx=0).stop 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 shutdown_connections 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3ecc0387c0 0x7f3ecc03ac80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 --2- 172.21.15.196:0/612428112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ee41011b0 0x7f3ee4195b60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 >> 172.21.15.196:0/612428112 conn(0x7f3ee40fab80 msgr2=0x7f3ee40fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 shutdown_connections 2024-03-26T22:02:36.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:36.788+0000 7f3ee93c1700 1 -- 172.21.15.196:0/612428112 wait complete. 2024-03-26T22:02:36.788 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:38.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:37 smithi194 ceph-mon[29693]: pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:38.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:37 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/612428112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:38.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:38.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:38.646 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:38.692 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:40.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:39 smithi194 ceph-mon[29693]: pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:40.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 -- 172.21.15.196:0/2710205894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 msgr2=0x7f8ef80ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:40.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 --2- 172.21.15.196:0/2710205894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80ffa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009910 tx=0x7f8ee8009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:40.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 -- 172.21.15.196:0/2710205894 shutdown_connections 2024-03-26T22:02:40.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 --2- 172.21.15.196:0/2710205894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80ffa40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 -- 172.21.15.196:0/2710205894 >> 172.21.15.196:0/2710205894 conn(0x7f8ef80fabc0 msgr2=0x7f8ef80fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 -- 172.21.15.196:0/2710205894 shutdown_connections 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.249+0000 7f8efc992700 1 -- 172.21.15.196:0/2710205894 wait complete. 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8efc992700 1 Processor -- start 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8efc992700 1 -- start start 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8efc992700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8efc992700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef8069a20 con 0x7f8ef80ff620 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8ef659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8ef659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44270/0 (socket says 172.21.15.196:44270) 2024-03-26T22:02:40.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.250+0000 7f8ef659c700 1 -- 172.21.15.196:0/942937585 learned_addr learned my addr 172.21.15.196:0/942937585 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8ef659c700 1 -- 172.21.15.196:0/942937585 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80095f0 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8ef659c700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80694e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8006010 tx=0x7f8ee8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee801d070 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ee8005dc0 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef8198920 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee8022e90 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.251+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef8198d60 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.252+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8ee800f4f0 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.252+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef818e450 con 0x7f8ef80ff620 2024-03-26T22:02:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.252+0000 7f8ee77fe700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8ee0038760 0x7f8ee003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.252+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8ee804cc40 con 0x7f8ef80ff620 2024-03-26T22:02:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.253+0000 7f8ef5d9b700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8ee0038760 0x7f8ee003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:40.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.253+0000 7f8ef5d9b700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8ee0038760 0x7f8ee003ac20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8eec006fd0 tx=0x7f8eec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:40.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.258+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8ee802a950 con 0x7f8ef80ff620 2024-03-26T22:02:40.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.527+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ef80623e0 con 0x7f8ef80ff620 2024-03-26T22:02:40.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.528+0000 7f8ee77fe700 1 -- 172.21.15.196:0/942937585 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8ee8027020 con 0x7f8ef80ff620 2024-03-26T22:02:40.527 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:40.527 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8ee0038760 msgr2=0x7f8ee003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8ee0038760 0x7f8ee003ac20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8eec006fd0 tx=0x7f8eec006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 msgr2=0x7f8ef80694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80694e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8006010 tx=0x7f8ee8009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 shutdown_connections 2024-03-26T22:02:40.529 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8ee0038760 0x7f8ee003ac20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:40.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 --2- 172.21.15.196:0/942937585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80ff620 0x7f8ef80694e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:40.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 >> 172.21.15.196:0/942937585 conn(0x7f8ef80fabc0 msgr2=0x7f8ef80fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:40.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 shutdown_connections 2024-03-26T22:02:40.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:40.530+0000 7f8efc992700 1 -- 172.21.15.196:0/942937585 wait complete. 2024-03-26T22:02:40.530 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:41.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:40 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/942937585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:42.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:41 smithi194 ceph-mon[29693]: pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:42.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:42.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:42.399 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:42.445 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.044+0000 7f3b10a46700 1 -- 172.21.15.196:0/1423013441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 msgr2=0x7f3b0c0ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.044+0000 7f3b10a46700 1 --2- 172.21.15.196:0/1423013441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c0ff1e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000aa20 tx=0x7f3b0000ad30 comp rx=0 tx=0).stop 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 -- 172.21.15.196:0/1423013441 shutdown_connections 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 --2- 172.21.15.196:0/1423013441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c0ff1e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 -- 172.21.15.196:0/1423013441 >> 172.21.15.196:0/1423013441 conn(0x7f3b0c0fa360 msgr2=0x7f3b0c0fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 -- 172.21.15.196:0/1423013441 shutdown_connections 2024-03-26T22:02:44.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 -- 172.21.15.196:0/1423013441 wait complete. 2024-03-26T22:02:44.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 Processor -- start 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 -- start start 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c195c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.045+0000 7f3b10a46700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1961d0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b0a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c195c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b0a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c195c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:46516/0 (socket says 172.21.15.196:46516) 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b0a59c700 1 -- 172.21.15.196:0/2798629768 learned_addr learned my addr 172.21.15.196:0/2798629768 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b0a59c700 1 -- 172.21.15.196:0/2798629768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0000a700 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b0a59c700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c195c90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000afd0 tx=0x7f3b00004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b0001d070 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0c1963d0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b000054a0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b0000e6e0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.046+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c196870 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.047+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3b00016470 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.047+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0c18f7b0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.047+0000 7f3afb7fe700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3af40387d0 0x7f3af403ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:44.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.047+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3b00012070 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.048+0000 7f3b09d9b700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3af40387d0 0x7f3af403ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:44.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.048+0000 7f3b09d9b700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3af40387d0 0x7f3af403ac90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3afc006fd0 tx=0x7f3afc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:44.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.053+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b00014370 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:43 smithi194 ceph-mon[29693]: pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:44.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.343+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b0c0623e0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.343+0000 7f3afb7fe700 1 -- 172.21.15.196:0/2798629768 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3b0c0623e0 con 0x7f3b0c0fedc0 2024-03-26T22:02:44.343 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:44.343 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3af40387d0 msgr2=0x7f3af403ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3af40387d0 0x7f3af403ac90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3afc006fd0 tx=0x7f3afc006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 msgr2=0x7f3b0c195c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c195c90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000afd0 tx=0x7f3b00004c00 comp rx=0 tx=0).stop 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 shutdown_connections 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3af40387d0 0x7f3af403ac90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 --2- 172.21.15.196:0/2798629768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b0c0fedc0 0x7f3b0c195c90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 >> 172.21.15.196:0/2798629768 conn(0x7f3b0c0fa360 msgr2=0x7f3b0c18a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:44.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 shutdown_connections 2024-03-26T22:02:44.346 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:44.346+0000 7f3b10a46700 1 -- 172.21.15.196:0/2798629768 wait complete. 2024-03-26T22:02:44.346 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:45.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:45 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2798629768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:45.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:45 smithi194 ceph-mon[29693]: pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:45.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:45.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:46.152 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:46.199 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 -- 172.21.15.196:0/4294426720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 msgr2=0x7feed80ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 --2- 172.21.15.196:0/4294426720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed80ffa50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7feec8009910 tx=0x7feec8009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 -- 172.21.15.196:0/4294426720 shutdown_connections 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 --2- 172.21.15.196:0/4294426720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed80ffa50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 -- 172.21.15.196:0/4294426720 >> 172.21.15.196:0/4294426720 conn(0x7feed80fabd0 msgr2=0x7feed80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 -- 172.21.15.196:0/4294426720 shutdown_connections 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 -- 172.21.15.196:0/4294426720 wait complete. 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.859+0000 7feedf890700 1 Processor -- start 2024-03-26T22:02:47.859 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedf890700 1 -- start start 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedf890700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed8069580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedf890700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed8069ac0 con 0x7feed80ff630 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedd62c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed8069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedd62c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed8069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:46522/0 (socket says 172.21.15.196:46522) 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedd62c700 1 -- 172.21.15.196:0/425127089 learned_addr learned my addr 172.21.15.196:0/425127089 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedd62c700 1 -- 172.21.15.196:0/425127089 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80095f0 con 0x7feed80ff630 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feedd62c700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed8069580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feec8006010 tx=0x7feec8005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.860+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec801d070 con 0x7feed80ff630 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.861+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feec800be20 con 0x7feed80ff630 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.861+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec8022e40 con 0x7feed80ff630 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.861+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed8198950 con 0x7feed80ff630 2024-03-26T22:02:47.860 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.861+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed8198df0 con 0x7feed80ff630 2024-03-26T22:02:47.861 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.862+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed818e460 con 0x7feed80ff630 2024-03-26T22:02:47.861 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.862+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7feec8022700 con 0x7feed80ff630 2024-03-26T22:02:47.861 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.862+0000 7feece7fc700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feec4038820 0x7feec403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:47.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.862+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feec804c760 con 0x7feed80ff630 2024-03-26T22:02:47.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.862+0000 7feedce2b700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feec4038820 0x7feec403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:47.862 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.863+0000 7feedce2b700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feec4038820 0x7feec403ace0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feed4006fd0 tx=0x7feed4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:47.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:47.867+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feec800b960 con 0x7feed80ff630 2024-03-26T22:02:48.140 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.141+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feed80623e0 con 0x7feed80ff630 2024-03-26T22:02:48.140 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.141+0000 7feece7fc700 1 -- 172.21.15.196:0/425127089 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7feed80623e0 con 0x7feed80ff630 2024-03-26T22:02:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:48.141 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.143+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feec4038820 msgr2=0x7feec403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.143+0000 7feedf890700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feec4038820 0x7feec403ace0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feed4006fd0 tx=0x7feed4006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.143+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 msgr2=0x7feed8069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.143+0000 7feedf890700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed8069580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feec8006010 tx=0x7feec8005ee0 comp rx=0 tx=0).stop 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.144+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 shutdown_connections 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.144+0000 7feedf890700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7feec4038820 0x7feec403ace0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.144+0000 7feedf890700 1 --2- 172.21.15.196:0/425127089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feed80ff630 0x7feed8069580 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.144+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 >> 172.21.15.196:0/425127089 conn(0x7feed80fabd0 msgr2=0x7feed80fb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.144+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 shutdown_connections 2024-03-26T22:02:48.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:48.144+0000 7feedf890700 1 -- 172.21.15.196:0/425127089 wait complete. 2024-03-26T22:02:48.144 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:48.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:47 smithi194 ceph-mon[29693]: pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:49.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:48 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/425127089' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:49.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:49.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:49.916 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:49.963 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:50.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:49 smithi194 ceph-mon[29693]: pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:51.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 -- 172.21.15.196:0/2344296875 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 msgr2=0x7f8670100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:51.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 --2- 172.21.15.196:0/2344296875 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f8670100fe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8660009910 tx=0x7f8660009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:51.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 -- 172.21.15.196:0/2344296875 shutdown_connections 2024-03-26T22:02:51.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 --2- 172.21.15.196:0/2344296875 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f8670100fe0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:51.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 -- 172.21.15.196:0/2344296875 >> 172.21.15.196:0/2344296875 conn(0x7f86700fa310 msgr2=0x7f86700fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:51.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 -- 172.21.15.196:0/2344296875 shutdown_connections 2024-03-26T22:02:51.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.535+0000 7f8675389700 1 -- 172.21.15.196:0/2344296875 wait complete. 2024-03-26T22:02:51.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f8675389700 1 Processor -- start 2024-03-26T22:02:51.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f8675389700 1 -- start start 2024-03-26T22:02:51.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f8675389700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f86700fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:51.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f8675389700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700fc6b0 con 0x7f8670100bc0 2024-03-26T22:02:51.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f866effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f86700fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:51.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f866effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f86700fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:46530/0 (socket says 172.21.15.196:46530) 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f866effd700 1 -- 172.21.15.196:0/2766750438 learned_addr learned my addr 172.21.15.196:0/2766750438 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.536+0000 7f866effd700 1 -- 172.21.15.196:0/2766750438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600095f0 con 0x7f8670100bc0 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.537+0000 7f866effd700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f86700fc170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8660006010 tx=0x7f8660009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.537+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f866001d070 con 0x7f8670100bc0 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.537+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86700fa940 con 0x7f8670100bc0 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.537+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8660005dc0 con 0x7f8670100bc0 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.537+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8660022e90 con 0x7f8670100bc0 2024-03-26T22:02:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.537+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700fade0 con 0x7f8670100bc0 2024-03-26T22:02:51.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.538+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8660022470 con 0x7f8670100bc0 2024-03-26T22:02:51.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.538+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86700fb7f0 con 0x7f8670100bc0 2024-03-26T22:02:51.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.538+0000 7f866cff9700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f865c038760 0x7f865c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:51.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.538+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f866004c8e0 con 0x7f8670100bc0 2024-03-26T22:02:51.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.538+0000 7f86667fc700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f865c038760 0x7f865c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:51.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.539+0000 7f86667fc700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f865c038760 0x7f865c03ac20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8658006fd0 tx=0x7f8658006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:51.542 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.543+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f866000b960 con 0x7f8670100bc0 2024-03-26T22:02:51.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.811+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8670062430 con 0x7f8670100bc0 2024-03-26T22:02:51.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.811+0000 7f866cff9700 1 -- 172.21.15.196:0/2766750438 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8660027020 con 0x7f8670100bc0 2024-03-26T22:02:51.811 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:51.811 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:51.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f865c038760 msgr2=0x7f865c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:51.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f865c038760 0x7f865c03ac20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8658006fd0 tx=0x7f8658006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:51.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 msgr2=0x7f86700fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:51.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f86700fc170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8660006010 tx=0x7f8660009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:51.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 shutdown_connections 2024-03-26T22:02:51.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f865c038760 0x7f865c03ac20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:51.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.813+0000 7f8675389700 1 --2- 172.21.15.196:0/2766750438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8670100bc0 0x7f86700fc170 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:51.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.814+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 >> 172.21.15.196:0/2766750438 conn(0x7f86700fa310 msgr2=0x7f86700fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:51.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.814+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 shutdown_connections 2024-03-26T22:02:51.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:51.814+0000 7f8675389700 1 -- 172.21.15.196:0/2766750438 wait complete. 2024-03-26T22:02:51.814 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:52.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:51 smithi194 ceph-mon[29693]: pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:53.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:52 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2766750438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:53.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:53.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:53.722 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:53.767 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:54.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:53 smithi194 ceph-mon[29693]: pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:55.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 -- 172.21.15.196:0/2012477137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 msgr2=0x7f4f500fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:55.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 --2- 172.21.15.196:0/2012477137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f500fee30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009910 tx=0x7f4f40009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:55.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 -- 172.21.15.196:0/2012477137 shutdown_connections 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 --2- 172.21.15.196:0/2012477137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f500fee30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 -- 172.21.15.196:0/2012477137 >> 172.21.15.196:0/2012477137 conn(0x7f4f500fa370 msgr2=0x7f4f500fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 -- 172.21.15.196:0/2012477137 shutdown_connections 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 -- 172.21.15.196:0/2012477137 wait complete. 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.382+0000 7f4f551ce700 1 Processor -- start 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f551ce700 1 -- start start 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f551ce700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f501012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f551ce700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50101800 con 0x7f4f500fea10 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f4ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f501012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f4ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f501012c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42278/0 (socket says 172.21.15.196:42278) 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f4ed9d700 1 -- 172.21.15.196:0/384277592 learned_addr learned my addr 172.21.15.196:0/384277592 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:55.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.383+0000 7f4f4ed9d700 1 -- 172.21.15.196:0/384277592 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f400095f0 con 0x7f4f500fea10 2024-03-26T22:02:55.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.384+0000 7f4f4ed9d700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f501012c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f40006010 tx=0x7f4f40009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.384+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4001d070 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.384+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f40005dc0 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.384+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f500ffa40 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.384+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f40022e90 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.384+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f500fff40 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.385+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f50106e30 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.385+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f4f4000b960 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.385+0000 7f4f37fff700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4f38038760 0x7f4f3803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.385+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f4004c780 con 0x7f4f500fea10 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.385+0000 7f4f4e59c700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4f38038760 0x7f4f3803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:55.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.386+0000 7f4f4e59c700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4f38038760 0x7f4f3803ac20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f44006fd0 tx=0x7f4f44006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:55.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.390+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f4000bd00 con 0x7f4f500fea10 2024-03-26T22:02:55.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:55 smithi194 ceph-mon[29693]: pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:55.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.659+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f500623e0 con 0x7f4f500fea10 2024-03-26T22:02:55.659 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.660+0000 7f4f37fff700 1 -- 172.21.15.196:0/384277592 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4f40027020 con 0x7f4f500fea10 2024-03-26T22:02:55.659 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:55.659 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.661+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4f38038760 msgr2=0x7f4f3803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.661+0000 7f4f551ce700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4f38038760 0x7f4f3803ac20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f44006fd0 tx=0x7f4f44006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.661+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 msgr2=0x7f4f501012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.661+0000 7f4f551ce700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f501012c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f40006010 tx=0x7f4f40009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.661+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 shutdown_connections 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.662+0000 7f4f551ce700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4f38038760 0x7f4f3803ac20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.662+0000 7f4f551ce700 1 --2- 172.21.15.196:0/384277592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f500fea10 0x7f4f501012c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.662+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 >> 172.21.15.196:0/384277592 conn(0x7f4f500fa370 msgr2=0x7f4f500fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.662+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 shutdown_connections 2024-03-26T22:02:55.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:55.662+0000 7f4f551ce700 1 -- 172.21.15.196:0/384277592 wait complete. 2024-03-26T22:02:55.662 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:02:56.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:56 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/384277592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:02:57.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:02:57.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:02:57.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:57 smithi194 ceph-mon[29693]: pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:02:57.552 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:57.598 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:02:59.142 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 -- 172.21.15.196:0/423461364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 msgr2=0x7fe924100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:59.142 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 --2- 172.21.15.196:0/423461364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924100b40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe920009910 tx=0x7fe920009c20 comp rx=0 tx=0).stop 2024-03-26T22:02:59.142 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 -- 172.21.15.196:0/423461364 shutdown_connections 2024-03-26T22:02:59.142 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 --2- 172.21.15.196:0/423461364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924100b40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 -- 172.21.15.196:0/423461364 >> 172.21.15.196:0/423461364 conn(0x7fe9240fbcc0 msgr2=0x7fe9240fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 -- 172.21.15.196:0/423461364 shutdown_connections 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.141+0000 7fe92bed7700 1 -- 172.21.15.196:0/423461364 wait complete. 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe92bed7700 1 Processor -- start 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe92bed7700 1 -- start start 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe92bed7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe92bed7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe924074f60 con 0x7fe924100720 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe929c73700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe929c73700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42292/0 (socket says 172.21.15.196:42292) 2024-03-26T22:02:59.143 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.142+0000 7fe929c73700 1 -- 172.21.15.196:0/1249664438 learned_addr learned my addr 172.21.15.196:0/1249664438 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:02:59.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe929c73700 1 -- 172.21.15.196:0/1249664438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200095f0 con 0x7fe924100720 2024-03-26T22:02:59.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe929c73700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924074a20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe920006010 tx=0x7fe920009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:59.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe92001d070 con 0x7fe924100720 2024-03-26T22:02:59.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240731a0 con 0x7fe924100720 2024-03-26T22:02:59.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe920005dc0 con 0x7fe924100720 2024-03-26T22:02:59.144 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe920022e90 con 0x7fe924100720 2024-03-26T22:02:59.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.143+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe924073640 con 0x7fe924100720 2024-03-26T22:02:59.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.144+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe92000b960 con 0x7fe924100720 2024-03-26T22:02:59.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.144+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe924106e80 con 0x7fe924100720 2024-03-26T22:02:59.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.144+0000 7fe91affd700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe910038760 0x7fe91003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:02:59.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.144+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe92004c8c0 con 0x7fe924100720 2024-03-26T22:02:59.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.144+0000 7fe929472700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe910038760 0x7fe91003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:02:59.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.145+0000 7fe929472700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe910038760 0x7fe91003ac20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe914006fd0 tx=0x7fe914006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:02:59.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.149+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe92000bd00 con 0x7fe924100720 2024-03-26T22:02:59.419 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.420+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9240623e0 con 0x7fe924100720 2024-03-26T22:02:59.419 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.420+0000 7fe91affd700 1 -- 172.21.15.196:0/1249664438 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe920027020 con 0x7fe924100720 2024-03-26T22:02:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:02:59.420 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:02:59.421 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe910038760 msgr2=0x7fe91003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe910038760 0x7fe91003ac20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe914006fd0 tx=0x7fe914006e40 comp rx=0 tx=0).stop 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 msgr2=0x7fe924074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924074a20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe920006010 tx=0x7fe920009f80 comp rx=0 tx=0).stop 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 shutdown_connections 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe910038760 0x7fe91003ac20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 --2- 172.21.15.196:0/1249664438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe924100720 0x7fe924074a20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 >> 172.21.15.196:0/1249664438 conn(0x7fe9240fbcc0 msgr2=0x7fe9240fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 shutdown_connections 2024-03-26T22:02:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:02:59.422+0000 7fe92bed7700 1 -- 172.21.15.196:0/1249664438 wait complete. 2024-03-26T22:02:59.423 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:00.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:59 smithi194 ceph-mon[29693]: pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:00.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:02:59 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1249664438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:01.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:01.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:01.208 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:01.253 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:02.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:01 smithi194 ceph-mon[29693]: pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:02.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 -- 172.21.15.196:0/515009613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 msgr2=0x7f51541013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:02.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 --2- 172.21.15.196:0/515009613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f51541013e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5150009910 tx=0x7f5150009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 -- 172.21.15.196:0/515009613 shutdown_connections 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 --2- 172.21.15.196:0/515009613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f51541013e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 -- 172.21.15.196:0/515009613 >> 172.21.15.196:0/515009613 conn(0x7f51540fabd0 msgr2=0x7f51540fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 -- 172.21.15.196:0/515009613 shutdown_connections 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.854+0000 7f515b940700 1 -- 172.21.15.196:0/515009613 wait complete. 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f515b940700 1 Processor -- start 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f515b940700 1 -- start start 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f515b940700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f515419c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f515b940700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515419c760 con 0x7f51540fefb0 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f51596dc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f515419c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f51596dc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f515419c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39228/0 (socket says 172.21.15.196:39228) 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.855+0000 7f51596dc700 1 -- 172.21.15.196:0/2213566968 learned_addr learned my addr 172.21.15.196:0/2213566968 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:02.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f51596dc700 1 -- 172.21.15.196:0/2213566968 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500095f0 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f51596dc700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f515419c220 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5150005b40 tx=0x7f5150005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f515001d070 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515419c960 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515419ce00 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5150005dc0 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.856+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5150022e10 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.857+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f515000f460 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.857+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5138005320 con 0x7f51540fefb0 2024-03-26T22:03:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.857+0000 7f514a7fc700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5140038770 0x7f514003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:02.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.857+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f515004c960 con 0x7f51540fefb0 2024-03-26T22:03:02.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.857+0000 7f5158edb700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5140038770 0x7f514003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:02.858 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.858+0000 7f5158edb700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5140038770 0x7f514003ac30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5144006fd0 tx=0x7f5144006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:02.861 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:02.862+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f515000f710 con 0x7f51540fefb0 2024-03-26T22:03:03.129 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.130+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5138005190 con 0x7f51540fefb0 2024-03-26T22:03:03.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.131+0000 7f514a7fc700 1 -- 172.21.15.196:0/2213566968 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5150027020 con 0x7f51540fefb0 2024-03-26T22:03:03.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:03.130 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:03.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5140038770 msgr2=0x7f514003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:03.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5140038770 0x7f514003ac30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5144006fd0 tx=0x7f5144006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 msgr2=0x7f515419c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f515419c220 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5150005b40 tx=0x7f5150005000 comp rx=0 tx=0).stop 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 shutdown_connections 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5140038770 0x7f514003ac30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 --2- 172.21.15.196:0/2213566968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51540fefb0 0x7f515419c220 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 >> 172.21.15.196:0/2213566968 conn(0x7f51540fabd0 msgr2=0x7f51540fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 shutdown_connections 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:03.133+0000 7f515b940700 1 -- 172.21.15.196:0/2213566968 wait complete. 2024-03-26T22:03:03.133 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:04.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:03 smithi194 ceph-mon[29693]: pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:04.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:03 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2213566968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:04.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:04.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:04.912 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:04.957 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:05.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:05 smithi194 ceph-mon[29693]: pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 -- 172.21.15.196:0/1730723277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 msgr2=0x7f2830103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 --2- 172.21.15.196:0/1730723277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830103640 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2820009910 tx=0x7f2820009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 -- 172.21.15.196:0/1730723277 shutdown_connections 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 --2- 172.21.15.196:0/1730723277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830103640 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 -- 172.21.15.196:0/1730723277 >> 172.21.15.196:0/1730723277 conn(0x7f28300fabc0 msgr2=0x7f28300fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 -- 172.21.15.196:0/1730723277 shutdown_connections 2024-03-26T22:03:06.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.491+0000 7f28367fe700 1 -- 172.21.15.196:0/1730723277 wait complete. 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.492+0000 7f28367fe700 1 Processor -- start 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.492+0000 7f28367fe700 1 -- start start 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.492+0000 7f28367fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.492+0000 7f28367fe700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301960d0 con 0x7f2830101250 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39244/0 (socket says 172.21.15.196:39244) 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282ffff700 1 -- 172.21.15.196:0/2332709330 learned_addr learned my addr 172.21.15.196:0/2332709330 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282ffff700 1 -- 172.21.15.196:0/2332709330 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28200095f0 con 0x7f2830101250 2024-03-26T22:03:06.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282ffff700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830195b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2820009ed0 tx=0x7f2820004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282001d070 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28301962d0 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f282000be20 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282000f460 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.493+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2830196770 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.494+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283018f550 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.494+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2820022800 con 0x7f2830101250 2024-03-26T22:03:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.495+0000 7f282d7fa700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28180387d0 0x7f281803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:06.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.495+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f282004cc00 con 0x7f2830101250 2024-03-26T22:03:06.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.495+0000 7f282f7fe700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28180387d0 0x7f281803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:06.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.495+0000 7f282f7fe700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28180387d0 0x7f281803ac90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2824006fd0 tx=0x7f2824006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:06.499 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.500+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2820053080 con 0x7f2830101250 2024-03-26T22:03:06.768 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.769+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28300623e0 con 0x7f2830101250 2024-03-26T22:03:06.768 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.769+0000 7f282d7fa700 1 -- 172.21.15.196:0/2332709330 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f28300623e0 con 0x7f2830101250 2024-03-26T22:03:06.769 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:06.769 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.771+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28180387d0 msgr2=0x7f281803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.771+0000 7f28367fe700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28180387d0 0x7f281803ac90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2824006fd0 tx=0x7f2824006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.771+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 msgr2=0x7f2830195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.771+0000 7f28367fe700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830195b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2820009ed0 tx=0x7f2820004690 comp rx=0 tx=0).stop 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.772+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 shutdown_connections 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.772+0000 7f28367fe700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f28180387d0 0x7f281803ac90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.772+0000 7f28367fe700 1 --2- 172.21.15.196:0/2332709330 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2830101250 0x7f2830195b90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.772+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 >> 172.21.15.196:0/2332709330 conn(0x7f28300fabc0 msgr2=0x7f28300fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.772+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 shutdown_connections 2024-03-26T22:03:06.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:06.772+0000 7f28367fe700 1 -- 172.21.15.196:0/2332709330 wait complete. 2024-03-26T22:03:06.772 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:08.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:07 smithi194 ceph-mon[29693]: pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:08.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:07 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2332709330' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:08.500 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:08.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:08.701 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:08.746 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:10.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:09 smithi194 ceph-mon[29693]: pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 -- 172.21.15.196:0/387193369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 msgr2=0x7f99c8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 --2- 172.21.15.196:0/387193369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8100b40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009910 tx=0x7f99c4009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 -- 172.21.15.196:0/387193369 shutdown_connections 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 --2- 172.21.15.196:0/387193369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8100b40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 -- 172.21.15.196:0/387193369 >> 172.21.15.196:0/387193369 conn(0x7f99c80fbcc0 msgr2=0x7f99c80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 -- 172.21.15.196:0/387193369 shutdown_connections 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.303+0000 7f99d0367700 1 -- 172.21.15.196:0/387193369 wait complete. 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99d0367700 1 Processor -- start 2024-03-26T22:03:10.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99d0367700 1 -- start start 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99d0367700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99d0367700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8074f60 con 0x7f99c8100720 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99ce103700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99ce103700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39270/0 (socket says 172.21.15.196:39270) 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99ce103700 1 -- 172.21.15.196:0/1624680657 learned_addr learned my addr 172.21.15.196:0/1624680657 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99ce103700 1 -- 172.21.15.196:0/1624680657 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c40095f0 con 0x7f99c8100720 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.304+0000 7f99ce103700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8074a20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f99c4006010 tx=0x7f99c4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.305+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99c401d070 con 0x7f99c8100720 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.305+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c80731a0 con 0x7f99c8100720 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.305+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99c4005dc0 con 0x7f99c8100720 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.305+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99c4022e90 con 0x7f99c8100720 2024-03-26T22:03:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.305+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c8073640 con 0x7f99c8100720 2024-03-26T22:03:10.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.305+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c8106e80 con 0x7f99c8100720 2024-03-26T22:03:10.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.306+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f99c4022470 con 0x7f99c8100720 2024-03-26T22:03:10.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.306+0000 7f99bf7fe700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f99b4038760 0x7f99b403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:10.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.306+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f99c404c7f0 con 0x7f99c8100720 2024-03-26T22:03:10.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.306+0000 7f99cd902700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f99b4038760 0x7f99b403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:10.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.307+0000 7f99cd902700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f99b4038760 0x7f99b403ac20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f99b8006fd0 tx=0x7f99b8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:10.310 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.311+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f99c400f4f0 con 0x7f99c8100720 2024-03-26T22:03:10.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.581+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99c818ef30 con 0x7f99c8100720 2024-03-26T22:03:10.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.581+0000 7f99bf7fe700 1 -- 172.21.15.196:0/1624680657 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f99c4027070 con 0x7f99c8100720 2024-03-26T22:03:10.581 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:10.581 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:10.582 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f99b4038760 msgr2=0x7f99b403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:10.582 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f99b4038760 0x7f99b403ac20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f99b8006fd0 tx=0x7f99b8006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 msgr2=0x7f99c8074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8074a20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f99c4006010 tx=0x7f99c4009f80 comp rx=0 tx=0).stop 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 shutdown_connections 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f99b4038760 0x7f99b403ac20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 --2- 172.21.15.196:0/1624680657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99c8100720 0x7f99c8074a20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 >> 172.21.15.196:0/1624680657 conn(0x7f99c80fbcc0 msgr2=0x7f99c80fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 shutdown_connections 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:10.583+0000 7f99d0367700 1 -- 172.21.15.196:0/1624680657 wait complete. 2024-03-26T22:03:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:10.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:10 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1624680657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:12.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:11 smithi194 ceph-mon[29693]: pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:12.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:12.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:12.377 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:12.422 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 -- 172.21.15.196:0/3648100802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 msgr2=0x7f31480ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 --2- 172.21.15.196:0/3648100802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f31480ffa30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f314000b0a0 tx=0x7f314000b3b0 comp rx=0 tx=0).stop 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 -- 172.21.15.196:0/3648100802 shutdown_connections 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 --2- 172.21.15.196:0/3648100802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f31480ffa30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 -- 172.21.15.196:0/3648100802 >> 172.21.15.196:0/3648100802 conn(0x7f31480fabb0 msgr2=0x7f31480fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 -- 172.21.15.196:0/3648100802 shutdown_connections 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.972+0000 7f314fde7700 1 -- 172.21.15.196:0/3648100802 wait complete. 2024-03-26T22:03:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314fde7700 1 Processor -- start 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314fde7700 1 -- start start 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314fde7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f3148195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314fde7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3148196140 con 0x7f31480ff610 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314db83700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f3148195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314db83700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f3148195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:37406/0 (socket says 172.21.15.196:37406) 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.973+0000 7f314db83700 1 -- 172.21.15.196:0/1729763650 learned_addr learned my addr 172.21.15.196:0/1729763650 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f314db83700 1 -- 172.21.15.196:0/1729763650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3140009d00 con 0x7f31480ff610 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f314db83700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f3148195c00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31400062a0 tx=0x7f314000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f314000e040 con 0x7f31480ff610 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3148196340 con 0x7f31480ff610 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3140003d70 con 0x7f31480ff610 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f314001c890 con 0x7f31480ff610 2024-03-26T22:03:13.976 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.974+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31481967e0 con 0x7f31480ff610 2024-03-26T22:03:13.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.975+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f314001cb50 con 0x7f31480ff610 2024-03-26T22:03:13.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.975+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314818f700 con 0x7f31480ff610 2024-03-26T22:03:13.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.975+0000 7f313effd700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31340387c0 0x7f313403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:13.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.975+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f314004c9a0 con 0x7f31480ff610 2024-03-26T22:03:13.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.975+0000 7f314d382700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31340387c0 0x7f313403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:13.977 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.976+0000 7f314d382700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31340387c0 0x7f313403ac80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3138006fd0 tx=0x7f3138006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:13.979 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:13.980+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f314001a3b0 con 0x7f31480ff610 2024-03-26T22:03:14.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:13 smithi194 ceph-mon[29693]: pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:14.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.252+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31480623e0 con 0x7f31480ff610 2024-03-26T22:03:14.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.252+0000 7f313effd700 1 -- 172.21.15.196:0/1729763650 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f314002be80 con 0x7f31480ff610 2024-03-26T22:03:14.252 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:14.252 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31340387c0 msgr2=0x7f313403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31340387c0 0x7f313403ac80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3138006fd0 tx=0x7f3138006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 msgr2=0x7f3148195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f3148195c00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31400062a0 tx=0x7f314000bd40 comp rx=0 tx=0).stop 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 shutdown_connections 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f31340387c0 0x7f313403ac80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 --2- 172.21.15.196:0/1729763650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31480ff610 0x7f3148195c00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:14.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 >> 172.21.15.196:0/1729763650 conn(0x7f31480fabb0 msgr2=0x7f3148068bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:14.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 shutdown_connections 2024-03-26T22:03:14.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:14.255+0000 7f314fde7700 1 -- 172.21.15.196:0/1729763650 wait complete. 2024-03-26T22:03:14.256 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:15.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:15 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1729763650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:15.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:15 smithi194 ceph-mon[29693]: pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:15.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:15.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:16.080 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:16.126 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:17.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 -- 172.21.15.196:0/903959044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 msgr2=0x7efe70100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 --2- 172.21.15.196:0/903959044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70100b40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efe60009910 tx=0x7efe60009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 -- 172.21.15.196:0/903959044 shutdown_connections 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 --2- 172.21.15.196:0/903959044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70100b40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 -- 172.21.15.196:0/903959044 >> 172.21.15.196:0/903959044 conn(0x7efe700fbcc0 msgr2=0x7efe700fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 -- 172.21.15.196:0/903959044 shutdown_connections 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.533+0000 7efe748c1700 1 -- 172.21.15.196:0/903959044 wait complete. 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe748c1700 1 Processor -- start 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe748c1700 1 -- start start 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe748c1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:17.535 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe748c1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe70196100 con 0x7efe70100720 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe6e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe6e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:37424/0 (socket says 172.21.15.196:37424) 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.534+0000 7efe6e59c700 1 -- 172.21.15.196:0/1916170290 learned_addr learned my addr 172.21.15.196:0/1916170290 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe6e59c700 1 -- 172.21.15.196:0/1916170290 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe600095f0 con 0x7efe70100720 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe6e59c700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70195bc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efe60006010 tx=0x7efe600050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6001d070 con 0x7efe70100720 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe70196300 con 0x7efe70100720 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe6000be20 con 0x7efe70100720 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe60022de0 con 0x7efe70100720 2024-03-26T22:03:17.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.535+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe701967a0 con 0x7efe70100720 2024-03-26T22:03:17.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.536+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7efe6000b960 con 0x7efe70100720 2024-03-26T22:03:17.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.536+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7018f550 con 0x7efe70100720 2024-03-26T22:03:17.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.536+0000 7efe5f7fe700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efe580387b0 0x7efe5803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:17.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.536+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efe6004c980 con 0x7efe70100720 2024-03-26T22:03:17.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.536+0000 7efe6dd9b700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efe580387b0 0x7efe5803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:17.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.537+0000 7efe6dd9b700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efe580387b0 0x7efe5803ac70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efe64006fd0 tx=0x7efe64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:17.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.541+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efe600226c0 con 0x7efe70100720 2024-03-26T22:03:17.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.808+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe700623e0 con 0x7efe70100720 2024-03-26T22:03:17.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.809+0000 7efe5f7fe700 1 -- 172.21.15.196:0/1916170290 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7efe700623e0 con 0x7efe70100720 2024-03-26T22:03:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:17.808 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:17.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.810+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efe580387b0 msgr2=0x7efe5803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:17.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.810+0000 7efe748c1700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efe580387b0 0x7efe5803ac70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efe64006fd0 tx=0x7efe64006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.810+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 msgr2=0x7efe70195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.810+0000 7efe748c1700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70195bc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efe60006010 tx=0x7efe600050b0 comp rx=0 tx=0).stop 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.811+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 shutdown_connections 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.811+0000 7efe748c1700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efe580387b0 0x7efe5803ac70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.811+0000 7efe748c1700 1 --2- 172.21.15.196:0/1916170290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efe70100720 0x7efe70195bc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.811+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 >> 172.21.15.196:0/1916170290 conn(0x7efe700fbcc0 msgr2=0x7efe700fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.811+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 shutdown_connections 2024-03-26T22:03:17.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:17.811+0000 7efe748c1700 1 -- 172.21.15.196:0/1916170290 wait complete. 2024-03-26T22:03:17.812 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:18.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:17 smithi194 ceph-mon[29693]: pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:19.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:18 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1916170290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:19.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:19.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:19.680 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:19.726 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:20.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:19 smithi194 ceph-mon[29693]: pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.262+0000 7f52fea36700 1 -- 172.21.15.196:0/509328350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 msgr2=0x7f52f80fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.262+0000 7f52fea36700 1 --2- 172.21.15.196:0/509328350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f80fee10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009910 tx=0x7f52ec009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.262+0000 7f52fea36700 1 -- 172.21.15.196:0/509328350 shutdown_connections 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.262+0000 7f52fea36700 1 --2- 172.21.15.196:0/509328350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f80fee10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.262+0000 7f52fea36700 1 -- 172.21.15.196:0/509328350 >> 172.21.15.196:0/509328350 conn(0x7f52f80fa370 msgr2=0x7f52f80fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.263+0000 7f52fea36700 1 -- 172.21.15.196:0/509328350 shutdown_connections 2024-03-26T22:03:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.263+0000 7f52fea36700 1 -- 172.21.15.196:0/509328350 wait complete. 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.263+0000 7f52fea36700 1 Processor -- start 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.263+0000 7f52fea36700 1 -- start start 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.263+0000 7f52fea36700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f8195c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.263+0000 7f52fea36700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8196150 con 0x7f52f80fe9f0 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f8195c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f8195c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:37446/0 (socket says 172.21.15.196:37446) 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f7fff700 1 -- 172.21.15.196:0/2682656610 learned_addr learned my addr 172.21.15.196:0/2682656610 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:21.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f7fff700 1 -- 172.21.15.196:0/2682656610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0095f0 con 0x7f52f80fe9f0 2024-03-26T22:03:21.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f7fff700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f8195c10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f52ec006010 tx=0x7f52ec0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:21.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ec01d070 con 0x7f52f80fe9f0 2024-03-26T22:03:21.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f8196350 con 0x7f52f80fe9f0 2024-03-26T22:03:21.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52ec00be20 con 0x7f52f80fe9f0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ec022de0 con 0x7f52f80fe9f0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.264+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f81967f0 con 0x7f52f80fe9f0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.265+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f818f570 con 0x7f52f80fe9f0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.265+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f52ec0226c0 con 0x7f52f80fe9f0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.265+0000 7f52f57fa700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f52e00387b0 0x7f52e003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.265+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f52ec04c730 con 0x7f52f80fe9f0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.266+0000 7f52f77fe700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f52e00387b0 0x7f52e003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:21.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.266+0000 7f52f77fe700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f52e00387b0 0x7f52e003ac70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006fd0 tx=0x7f52e8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:21.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.271+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f52ec022470 con 0x7f52f80fe9f0 2024-03-26T22:03:21.538 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.539+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52f80ff520 con 0x7f52f80fe9f0 2024-03-26T22:03:21.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.539+0000 7f52f57fa700 1 -- 172.21.15.196:0/2682656610 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f52f80ff520 con 0x7f52f80fe9f0 2024-03-26T22:03:21.539 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:21.539 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.541+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f52e00387b0 msgr2=0x7f52e003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.541+0000 7f52fea36700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f52e00387b0 0x7f52e003ac70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006fd0 tx=0x7f52e8006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.541+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 msgr2=0x7f52f8195c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.541+0000 7f52fea36700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f8195c10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f52ec006010 tx=0x7f52ec0050b0 comp rx=0 tx=0).stop 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.542+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 shutdown_connections 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.542+0000 7f52fea36700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f52e00387b0 0x7f52e003ac70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.542+0000 7f52fea36700 1 --2- 172.21.15.196:0/2682656610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f80fe9f0 0x7f52f8195c10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.542+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 >> 172.21.15.196:0/2682656610 conn(0x7f52f80fa370 msgr2=0x7f52f80fadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:21.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.542+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 shutdown_connections 2024-03-26T22:03:21.542 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:21.542+0000 7f52fea36700 1 -- 172.21.15.196:0/2682656610 wait complete. 2024-03-26T22:03:21.542 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:22.166 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:21 smithi194 ceph-mon[29693]: pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:22.166 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:21 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2682656610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:23.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:23.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:23.415 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:23.461 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:24.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:23 smithi194 ceph-mon[29693]: pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:25.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 -- 172.21.15.196:0/995335318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 msgr2=0x7fe930100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:25.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 --2- 172.21.15.196:0/995335318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930100b60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe924009910 tx=0x7fe924009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:25.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 -- 172.21.15.196:0/995335318 shutdown_connections 2024-03-26T22:03:25.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 --2- 172.21.15.196:0/995335318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930100b60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 -- 172.21.15.196:0/995335318 >> 172.21.15.196:0/995335318 conn(0x7fe9300fbd00 msgr2=0x7fe9300fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 -- 172.21.15.196:0/995335318 shutdown_connections 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.035+0000 7fe934dfc700 1 -- 172.21.15.196:0/995335318 wait complete. 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe934dfc700 1 Processor -- start 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe934dfc700 1 -- start start 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe934dfc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe934dfc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930074f60 con 0x7fe930100740 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe92e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe92e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42102/0 (socket says 172.21.15.196:42102) 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe92e59c700 1 -- 172.21.15.196:0/2600348498 learned_addr learned my addr 172.21.15.196:0/2600348498 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.036+0000 7fe92e59c700 1 -- 172.21.15.196:0/2600348498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240095f0 con 0x7fe930100740 2024-03-26T22:03:25.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.037+0000 7fe92e59c700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930074a20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe924006010 tx=0x7fe924009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.037+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe92401d070 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.037+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9300731a0 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.037+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe924005dc0 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.037+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe924022e90 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.037+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe930073640 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.038+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe930106ea0 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.038+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe924022470 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.038+0000 7fe91f7fe700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe918038760 0x7fe91803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.038+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe92404c7f0 con 0x7fe930100740 2024-03-26T22:03:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.038+0000 7fe92dd9b700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe918038760 0x7fe91803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:25.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.039+0000 7fe92dd9b700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe918038760 0x7fe91803ac20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe920006fd0 tx=0x7fe920006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:25.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.043+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe92400b960 con 0x7fe930100740 2024-03-26T22:03:25.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.311+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9300623e0 con 0x7fe930100740 2024-03-26T22:03:25.312 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.312+0000 7fe91f7fe700 1 -- 172.21.15.196:0/2600348498 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe924027030 con 0x7fe930100740 2024-03-26T22:03:25.312 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:25.312 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:25.313 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe918038760 msgr2=0x7fe91803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe918038760 0x7fe91803ac20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe920006fd0 tx=0x7fe920006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 msgr2=0x7fe930074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930074a20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe924006010 tx=0x7fe924009f80 comp rx=0 tx=0).stop 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 shutdown_connections 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe918038760 0x7fe91803ac20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 --2- 172.21.15.196:0/2600348498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe930100740 0x7fe930074a20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 >> 172.21.15.196:0/2600348498 conn(0x7fe9300fbd00 msgr2=0x7fe9300fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 shutdown_connections 2024-03-26T22:03:25.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:25.314+0000 7fe934dfc700 1 -- 172.21.15.196:0/2600348498 wait complete. 2024-03-26T22:03:25.315 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:25.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:25 smithi194 ceph-mon[29693]: pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:26.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:26 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2600348498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:26.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:26.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:27.137 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:27.182 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:27.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:27 smithi194 ceph-mon[29693]: pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 -- 172.21.15.196:0/2455721192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 msgr2=0x7f44700ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 --2- 172.21.15.196:0/2455721192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f44700ffa50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f446800b0a0 tx=0x7f446800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 -- 172.21.15.196:0/2455721192 shutdown_connections 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 --2- 172.21.15.196:0/2455721192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f44700ffa50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 -- 172.21.15.196:0/2455721192 >> 172.21.15.196:0/2455721192 conn(0x7f44700fabd0 msgr2=0x7f44700fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 -- 172.21.15.196:0/2455721192 shutdown_connections 2024-03-26T22:03:28.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.804+0000 7f4476c54700 1 -- 172.21.15.196:0/2455721192 wait complete. 2024-03-26T22:03:28.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f4476c54700 1 Processor -- start 2024-03-26T22:03:28.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f4476c54700 1 -- start start 2024-03-26T22:03:28.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f4476c54700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f4470195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:28.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f4476c54700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4470196190 con 0x7f44700ff630 2024-03-26T22:03:28.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f44749f0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f4470195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:28.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f44749f0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f4470195c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:42122/0 (socket says 172.21.15.196:42122) 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f44749f0700 1 -- 172.21.15.196:0/2032116179 learned_addr learned my addr 172.21.15.196:0/2032116179 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.805+0000 7f44749f0700 1 -- 172.21.15.196:0/2032116179 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468009d00 con 0x7f44700ff630 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.806+0000 7f44749f0700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f4470195c50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f44680062a0 tx=0x7f446800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.806+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4468020070 con 0x7f44700ff630 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.806+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4470196390 con 0x7f44700ff630 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.806+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4468003cd0 con 0x7f44700ff630 2024-03-26T22:03:28.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.806+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f446800e040 con 0x7f44700ff630 2024-03-26T22:03:28.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.806+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4470196830 con 0x7f44700ff630 2024-03-26T22:03:28.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.807+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f4468004040 con 0x7f44700ff630 2024-03-26T22:03:28.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.807+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447018f780 con 0x7f44700ff630 2024-03-26T22:03:28.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.807+0000 7f446dffb700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f445c0387b0 0x7f445c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:28.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.807+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f446804c0c0 con 0x7f44700ff630 2024-03-26T22:03:28.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.807+0000 7f446ffff700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f445c0387b0 0x7f445c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:28.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.808+0000 7f446ffff700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f445c0387b0 0x7f445c03ac70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4460006fd0 tx=0x7f4460006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:28.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:28.812+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4468011dd0 con 0x7f44700ff630 2024-03-26T22:03:29.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.092+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44700623e0 con 0x7f44700ff630 2024-03-26T22:03:29.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.093+0000 7f446dffb700 1 -- 172.21.15.196:0/2032116179 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f44700623e0 con 0x7f44700ff630 2024-03-26T22:03:29.093 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:29.094 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:29.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f445c0387b0 msgr2=0x7f445c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:29.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f445c0387b0 0x7f445c03ac70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4460006fd0 tx=0x7f4460006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:29.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 msgr2=0x7f4470195c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:29.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f4470195c50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f44680062a0 tx=0x7f446800bd40 comp rx=0 tx=0).stop 2024-03-26T22:03:29.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 shutdown_connections 2024-03-26T22:03:29.095 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f445c0387b0 0x7f445c03ac70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:29.096 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 --2- 172.21.15.196:0/2032116179 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44700ff630 0x7f4470195c50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:29.096 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 >> 172.21.15.196:0/2032116179 conn(0x7f44700fabd0 msgr2=0x7f4470068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:29.096 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 shutdown_connections 2024-03-26T22:03:29.096 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:29.095+0000 7f4476c54700 1 -- 172.21.15.196:0/2032116179 wait complete. 2024-03-26T22:03:29.096 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:30.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:29 smithi194 ceph-mon[29693]: pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:30.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:29 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2032116179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:30.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:30.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:30.983 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:31.029 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:32.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:31 smithi194 ceph-mon[29693]: pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:32.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 -- 172.21.15.196:0/3196874914 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 msgr2=0x7f02e00fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:32.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 --2- 172.21.15.196:0/3196874914 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e00fee20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009910 tx=0x7f02d4009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:32.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 -- 172.21.15.196:0/3196874914 shutdown_connections 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 --2- 172.21.15.196:0/3196874914 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e00fee20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 -- 172.21.15.196:0/3196874914 >> 172.21.15.196:0/3196874914 conn(0x7f02e00fa360 msgr2=0x7f02e00fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 -- 172.21.15.196:0/3196874914 shutdown_connections 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.601+0000 7f02e6eaa700 1 -- 172.21.15.196:0/3196874914 wait complete. 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e6eaa700 1 Processor -- start 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e6eaa700 1 -- start start 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e6eaa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e0101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e6eaa700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e0101750 con 0x7f02e00fea00 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e4c46700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e0101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e4c46700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e0101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44964/0 (socket says 172.21.15.196:44964) 2024-03-26T22:03:32.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e4c46700 1 -- 172.21.15.196:0/4029875921 learned_addr learned my addr 172.21.15.196:0/4029875921 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:32.611 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.602+0000 7f02e4c46700 1 -- 172.21.15.196:0/4029875921 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40095f0 con 0x7f02e00fea00 2024-03-26T22:03:32.611 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.603+0000 7f02e4c46700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e0101210 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f02d4006010 tx=0x7f02d4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:32.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.603+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02d401d070 con 0x7f02e00fea00 2024-03-26T22:03:32.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.603+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e00ff990 con 0x7f02e00fea00 2024-03-26T22:03:32.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.603+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e00ffe30 con 0x7f02e00fea00 2024-03-26T22:03:32.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.603+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02d4005dc0 con 0x7f02e00fea00 2024-03-26T22:03:32.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.603+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02d4022e70 con 0x7f02e00fea00 2024-03-26T22:03:32.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.604+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e0106eb0 con 0x7f02e00fea00 2024-03-26T22:03:32.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.604+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f02d400b960 con 0x7f02e00fea00 2024-03-26T22:03:32.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.604+0000 7f02ddffb700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f02d0038770 0x7f02d003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:32.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.604+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f02d404c710 con 0x7f02e00fea00 2024-03-26T22:03:32.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.604+0000 7f02dffff700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f02d0038770 0x7f02d003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:32.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.605+0000 7f02dffff700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f02d0038770 0x7f02d003ac30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02cc006fd0 tx=0x7f02cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:32.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.609+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f02d4022870 con 0x7f02e00fea00 2024-03-26T22:03:32.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.877+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02e00ff530 con 0x7f02e00fea00 2024-03-26T22:03:32.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.878+0000 7f02ddffb700 1 -- 172.21.15.196:0/4029875921 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f02e00ff530 con 0x7f02e00fea00 2024-03-26T22:03:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:32.877 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f02d0038770 msgr2=0x7f02d003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f02d0038770 0x7f02d003ac30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02cc006fd0 tx=0x7f02cc006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 msgr2=0x7f02e0101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e0101210 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f02d4006010 tx=0x7f02d4009f80 comp rx=0 tx=0).stop 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 shutdown_connections 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f02d0038770 0x7f02d003ac30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 --2- 172.21.15.196:0/4029875921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02e00fea00 0x7f02e0101210 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 >> 172.21.15.196:0/4029875921 conn(0x7f02e00fa360 msgr2=0x7f02e00fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 shutdown_connections 2024-03-26T22:03:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:32.880+0000 7f02e6eaa700 1 -- 172.21.15.196:0/4029875921 wait complete. 2024-03-26T22:03:32.880 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:33 smithi194 ceph-mon[29693]: pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:34.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:33 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4029875921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:34.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:34.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:34.695 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:34.741 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:35.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:35 smithi194 ceph-mon[29693]: pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 -- 172.21.15.196:0/1940674029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 msgr2=0x7f8c100ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 --2- 172.21.15.196:0/1940674029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c100ffa40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009910 tx=0x7f8c00009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 -- 172.21.15.196:0/1940674029 shutdown_connections 2024-03-26T22:03:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 --2- 172.21.15.196:0/1940674029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c100ffa40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 -- 172.21.15.196:0/1940674029 >> 172.21.15.196:0/1940674029 conn(0x7f8c100fab80 msgr2=0x7f8c100fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 -- 172.21.15.196:0/1940674029 shutdown_connections 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.122+0000 7f8c1687d700 1 -- 172.21.15.196:0/1940674029 wait complete. 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.123+0000 7f8c1687d700 1 Processor -- start 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.123+0000 7f8c1687d700 1 -- start start 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.123+0000 7f8c1687d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c10195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.123+0000 7f8c1687d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c10196130 con 0x7f8c100ff620 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c10195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c10195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44990/0 (socket says 172.21.15.196:44990) 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0ffff700 1 -- 172.21.15.196:0/3583808768 learned_addr learned my addr 172.21.15.196:0/3583808768 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0ffff700 1 -- 172.21.15.196:0/3583808768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c000095f0 con 0x7f8c100ff620 2024-03-26T22:03:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0ffff700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c10195bf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8c00006010 tx=0x7f8c00009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:36.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c0001d070 con 0x7f8c100ff620 2024-03-26T22:03:36.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c10196330 con 0x7f8c100ff620 2024-03-26T22:03:36.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c00005dc0 con 0x7f8c100ff620 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c101967d0 con 0x7f8c100ff620 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.124+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c00022e90 con 0x7f8c100ff620 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.125+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1018f720 con 0x7f8c100ff620 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.125+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8c00022470 con 0x7f8c100ff620 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.125+0000 7f8c0d7fa700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8bf8038760 0x7f8bf803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.125+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8c0004c8c0 con 0x7f8c100ff620 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.126+0000 7f8c0f7fe700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8bf8038760 0x7f8bf803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.126+0000 7f8c0f7fe700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8bf8038760 0x7f8bf803ac20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8c04006fd0 tx=0x7f8c04006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:36.129 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.130+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c0000f510 con 0x7f8c100ff620 2024-03-26T22:03:36.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.399+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c100623e0 con 0x7f8c100ff620 2024-03-26T22:03:36.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.400+0000 7f8c0d7fa700 1 -- 172.21.15.196:0/3583808768 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8c00027030 con 0x7f8c100ff620 2024-03-26T22:03:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:36.400 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:36.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8bf8038760 msgr2=0x7f8bf803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8bf8038760 0x7f8bf803ac20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8c04006fd0 tx=0x7f8c04006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 msgr2=0x7f8c10195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c10195bf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8c00006010 tx=0x7f8c00009f80 comp rx=0 tx=0).stop 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 shutdown_connections 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8bf8038760 0x7f8bf803ac20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 --2- 172.21.15.196:0/3583808768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c100ff620 0x7f8c10195bf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 >> 172.21.15.196:0/3583808768 conn(0x7f8c100fab80 msgr2=0x7f8c100682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 shutdown_connections 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:36.402+0000 7f8c1687d700 1 -- 172.21.15.196:0/3583808768 wait complete. 2024-03-26T22:03:36.403 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:36.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:36 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3583808768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:37.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:37 smithi194 ceph-mon[29693]: pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:38.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:38.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:38.276 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:38.321 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:39.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 -- 172.21.15.196:0/1588153122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 msgr2=0x7f2da0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 --2- 172.21.15.196:0/1588153122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0100b40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009910 tx=0x7f2d94009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 -- 172.21.15.196:0/1588153122 shutdown_connections 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 --2- 172.21.15.196:0/1588153122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0100b40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 -- 172.21.15.196:0/1588153122 >> 172.21.15.196:0/1588153122 conn(0x7f2da00fbcc0 msgr2=0x7f2da00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 -- 172.21.15.196:0/1588153122 shutdown_connections 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.841+0000 7f2da70b4700 1 -- 172.21.15.196:0/1588153122 wait complete. 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.842+0000 7f2da70b4700 1 Processor -- start 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.842+0000 7f2da70b4700 1 -- start start 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.842+0000 7f2da70b4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.842+0000 7f2da70b4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da0196100 con 0x7f2da0100720 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2da4e50700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2da4e50700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45004/0 (socket says 172.21.15.196:45004) 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2da4e50700 1 -- 172.21.15.196:0/4054029718 learned_addr learned my addr 172.21.15.196:0/4054029718 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2da4e50700 1 -- 172.21.15.196:0/4054029718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d940095f0 con 0x7f2da0100720 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2da4e50700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0195bc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2d94006010 tx=0x7f2d940050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9401d070 con 0x7f2da0100720 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da0196300 con 0x7f2da0100720 2024-03-26T22:03:39.844 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d9400be20 con 0x7f2da0100720 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.843+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d94022de0 con 0x7f2da0100720 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.844+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da01967a0 con 0x7f2da0100720 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.844+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2d9400b960 con 0x7f2da0100720 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.844+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da018f550 con 0x7f2da0100720 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.845+0000 7f2d9dffb700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d900387b0 0x7f2d9003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.845+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2d9404c8c0 con 0x7f2da0100720 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.845+0000 7f2d9ffff700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d900387b0 0x7f2d9003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:39.845 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.845+0000 7f2d9ffff700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d900387b0 0x7f2d9003ac70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c006fd0 tx=0x7f2d8c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:39.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:39.850+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2d940226c0 con 0x7f2da0100720 2024-03-26T22:03:40.121 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.122+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2da00623e0 con 0x7f2da0100720 2024-03-26T22:03:40.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.123+0000 7f2d9dffb700 1 -- 172.21.15.196:0/4054029718 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2da00623e0 con 0x7f2da0100720 2024-03-26T22:03:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:40.122 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:40.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d900387b0 msgr2=0x7f2d9003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:40.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d900387b0 0x7f2d9003ac70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c006fd0 tx=0x7f2d8c006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:40.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 msgr2=0x7f2da0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:40.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0195bc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2d94006010 tx=0x7f2d940050b0 comp rx=0 tx=0).stop 2024-03-26T22:03:40.124 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 shutdown_connections 2024-03-26T22:03:40.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2d900387b0 0x7f2d9003ac70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:40.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 --2- 172.21.15.196:0/4054029718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2da0100720 0x7f2da0195bc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:40.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 >> 172.21.15.196:0/4054029718 conn(0x7f2da00fbcc0 msgr2=0x7f2da00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:40.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 shutdown_connections 2024-03-26T22:03:40.125 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:40.125+0000 7f2da70b4700 1 -- 172.21.15.196:0/4054029718 wait complete. 2024-03-26T22:03:40.125 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:40.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:39 smithi194 ceph-mon[29693]: pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:40.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:40 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4054029718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:41.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:41.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:42.072 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:42.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:41 smithi194 ceph-mon[29693]: pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 -- 172.21.15.196:0/3448877211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 msgr2=0x7f7584100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 --2- 172.21.15.196:0/3448877211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584100b60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f757c00b0a0 tx=0x7f757c00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 -- 172.21.15.196:0/3448877211 shutdown_connections 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 --2- 172.21.15.196:0/3448877211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584100b60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 -- 172.21.15.196:0/3448877211 >> 172.21.15.196:0/3448877211 conn(0x7f75840fbd00 msgr2=0x7f75840fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 -- 172.21.15.196:0/3448877211 shutdown_connections 2024-03-26T22:03:43.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.638+0000 7f758950d700 1 -- 172.21.15.196:0/3448877211 wait complete. 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.639+0000 7f758950d700 1 Processor -- start 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.639+0000 7f758950d700 1 -- start start 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.639+0000 7f758950d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.639+0000 7f758950d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75841960d0 con 0x7f7584100740 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f7582ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f7582ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:37922/0 (socket says 172.21.15.196:37922) 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f7582ffd700 1 -- 172.21.15.196:0/298666532 learned_addr learned my addr 172.21.15.196:0/298666532 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f7582ffd700 1 -- 172.21.15.196:0/298666532 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c009d00 con 0x7f7584100740 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f7582ffd700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584195b90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f757c0062a0 tx=0x7f757c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c00e040 con 0x7f7584100740 2024-03-26T22:03:43.640 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75841962d0 con 0x7f7584100740 2024-03-26T22:03:43.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f757c003d70 con 0x7f7584100740 2024-03-26T22:03:43.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c01c890 con 0x7f7584100740 2024-03-26T22:03:43.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.640+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7584196770 con 0x7f7584100740 2024-03-26T22:03:43.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.641+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758418f570 con 0x7f7584100740 2024-03-26T22:03:43.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.641+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f757c01cb50 con 0x7f7584100740 2024-03-26T22:03:43.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.642+0000 7f756bfff700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f756c0387c0 0x7f756c03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:43.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.642+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f757c04c940 con 0x7f7584100740 2024-03-26T22:03:43.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.642+0000 7f75827fc700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f756c0387c0 0x7f756c03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:43.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.642+0000 7f75827fc700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f756c0387c0 0x7f756c03ac80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7578006fd0 tx=0x7f7578006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:43.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.647+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f757c01a590 con 0x7f7584100740 2024-03-26T22:03:43.913 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.914+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75840623e0 con 0x7f7584100740 2024-03-26T22:03:43.914 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.915+0000 7f756bfff700 1 -- 172.21.15.196:0/298666532 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f75840623e0 con 0x7f7584100740 2024-03-26T22:03:43.914 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:43.914 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f756c0387c0 msgr2=0x7f756c03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f756c0387c0 0x7f756c03ac80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7578006fd0 tx=0x7f7578006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 msgr2=0x7f7584195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584195b90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f757c0062a0 tx=0x7f757c00bd40 comp rx=0 tx=0).stop 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 shutdown_connections 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f756c0387c0 0x7f756c03ac80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 --2- 172.21.15.196:0/298666532 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7584100740 0x7f7584195b90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 >> 172.21.15.196:0/298666532 conn(0x7f75840fbd00 msgr2=0x7f75840fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 shutdown_connections 2024-03-26T22:03:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:43.917+0000 7f758950d700 1 -- 172.21.15.196:0/298666532 wait complete. 2024-03-26T22:03:43.918 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:44.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:43 smithi194 ceph-mon[29693]: pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:45.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:44 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/298666532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:45.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:45.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:45.748 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:45.794 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:45.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:45 smithi194 ceph-mon[29693]: pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 -- 172.21.15.196:0/2444424709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 msgr2=0x7f0800100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 --2- 172.21.15.196:0/2444424709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800100b30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f07fc009910 tx=0x7f07fc009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 -- 172.21.15.196:0/2444424709 shutdown_connections 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 --2- 172.21.15.196:0/2444424709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800100b30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 -- 172.21.15.196:0/2444424709 >> 172.21.15.196:0/2444424709 conn(0x7f08000fbcd0 msgr2=0x7f08000fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 -- 172.21.15.196:0/2444424709 shutdown_connections 2024-03-26T22:03:47.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.281+0000 7f0807c08700 1 -- 172.21.15.196:0/2444424709 wait complete. 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f0807c08700 1 Processor -- start 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f0807c08700 1 -- start start 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f0807c08700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f0807c08700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0800196120 con 0x7f0800100710 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f08059a4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f08059a4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:37948/0 (socket says 172.21.15.196:37948) 2024-03-26T22:03:47.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f08059a4700 1 -- 172.21.15.196:0/2317934240 learned_addr learned my addr 172.21.15.196:0/2317934240 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f08059a4700 1 -- 172.21.15.196:0/2317934240 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0095f0 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.282+0000 7f08059a4700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800195be0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f07fc006010 tx=0x7f07fc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.283+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07fc01d070 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.283+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0800196320 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.283+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07fc005dc0 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.283+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08001967c0 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.283+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07fc022e50 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.284+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f07fc022470 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.284+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080018f540 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.284+0000 7f07f6ffd700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07ec038770 0x7f07ec03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.284+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f07fc04c7d0 con 0x7f0800100710 2024-03-26T22:03:47.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.284+0000 7f08051a3700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07ec038770 0x7f07ec03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:47.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.284+0000 7f08051a3700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07ec038770 0x7f07ec03ac30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f07f0006fd0 tx=0x7f07f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:47.288 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.289+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07fc00f460 con 0x7f0800100710 2024-03-26T22:03:47.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.555+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08000623e0 con 0x7f0800100710 2024-03-26T22:03:47.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.555+0000 7f07f6ffd700 1 -- 172.21.15.196:0/2317934240 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f08000623e0 con 0x7f0800100710 2024-03-26T22:03:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:47.555 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07ec038770 msgr2=0x7f07ec03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07ec038770 0x7f07ec03ac30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f07f0006fd0 tx=0x7f07f0006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 msgr2=0x7f0800195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800195be0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f07fc006010 tx=0x7f07fc009f80 comp rx=0 tx=0).stop 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 shutdown_connections 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f07ec038770 0x7f07ec03ac30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 --2- 172.21.15.196:0/2317934240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0800100710 0x7f0800195be0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 >> 172.21.15.196:0/2317934240 conn(0x7f08000fbcd0 msgr2=0x7f08000fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 shutdown_connections 2024-03-26T22:03:47.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:47.557+0000 7f0807c08700 1 -- 172.21.15.196:0/2317934240 wait complete. 2024-03-26T22:03:47.558 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:48.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:47 smithi194 ceph-mon[29693]: pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:48.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:47 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2317934240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:49.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:49.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:49.430 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:49.475 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:50.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:49 smithi194 ceph-mon[29693]: pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:50.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 -- 172.21.15.196:0/3178276864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 msgr2=0x7f5f7c101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 --2- 172.21.15.196:0/3178276864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c101030 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009910 tx=0x7f5f78009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 -- 172.21.15.196:0/3178276864 shutdown_connections 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 --2- 172.21.15.196:0/3178276864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c101030 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 -- 172.21.15.196:0/3178276864 >> 172.21.15.196:0/3178276864 conn(0x7f5f7c0fa370 msgr2=0x7f5f7c0fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 -- 172.21.15.196:0/3178276864 shutdown_connections 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.980+0000 7f5f84486700 1 -- 172.21.15.196:0/3178276864 wait complete. 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f84486700 1 Processor -- start 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f84486700 1 -- start start 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f84486700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:50.981 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f84486700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c196110 con 0x7f5f7c100c10 2024-03-26T22:03:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f82222700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f82222700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:37966/0 (socket says 172.21.15.196:37966) 2024-03-26T22:03:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f82222700 1 -- 172.21.15.196:0/1247524446 learned_addr learned my addr 172.21.15.196:0/1247524446 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.981+0000 7f5f82222700 1 -- 172.21.15.196:0/1247524446 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780095f0 con 0x7f5f7c100c10 2024-03-26T22:03:50.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.982+0000 7f5f82222700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c195bd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5f78006010 tx=0x7f5f78009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:50.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.982+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7801d070 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.982+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f78005dc0 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.982+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c196310 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.982+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f78022e90 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.982+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c1967b0 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.983+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c18f730 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.983+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f5f78022470 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.983+0000 7f5f737fe700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5f68038760 0x7f5f6803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.983+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f7804c870 con 0x7f5f7c100c10 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.983+0000 7f5f81a21700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5f68038760 0x7f5f6803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.984+0000 7f5f81a21700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5f68038760 0x7f5f6803ac20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c006fd0 tx=0x7f5f6c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:50.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:50.988+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5f7800baf0 con 0x7f5f7c100c10 2024-03-26T22:03:51.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.258+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f7c101740 con 0x7f5f7c100c10 2024-03-26T22:03:51.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.259+0000 7f5f737fe700 1 -- 172.21.15.196:0/1247524446 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5f7c101740 con 0x7f5f7c100c10 2024-03-26T22:03:51.258 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:51.258 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:51.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5f68038760 msgr2=0x7f5f6803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:51.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5f68038760 0x7f5f6803ac20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c006fd0 tx=0x7f5f6c006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:51.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 msgr2=0x7f5f7c195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c195bd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5f78006010 tx=0x7f5f78009f80 comp rx=0 tx=0).stop 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 shutdown_connections 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5f68038760 0x7f5f6803ac20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 --2- 172.21.15.196:0/1247524446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f7c100c10 0x7f5f7c195bd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 >> 172.21.15.196:0/1247524446 conn(0x7f5f7c0fa370 msgr2=0x7f5f7c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 shutdown_connections 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:51.261+0000 7f5f84486700 1 -- 172.21.15.196:0/1247524446 wait complete. 2024-03-26T22:03:51.261 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:52.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:51 smithi194 ceph-mon[29693]: pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:52.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:51 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1247524446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:52.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:52.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:53.040 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:53.085 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:54.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:53 smithi194 ceph-mon[29693]: pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:54.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 -- 172.21.15.196:0/3687731854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 msgr2=0x7fdf300ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 --2- 172.21.15.196:0/3687731854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf300ffa30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800b0a0 tx=0x7fdf2800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 -- 172.21.15.196:0/3687731854 shutdown_connections 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 --2- 172.21.15.196:0/3687731854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf300ffa30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 -- 172.21.15.196:0/3687731854 >> 172.21.15.196:0/3687731854 conn(0x7fdf300fabb0 msgr2=0x7fdf300fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 -- 172.21.15.196:0/3687731854 shutdown_connections 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 -- 172.21.15.196:0/3687731854 wait complete. 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 Processor -- start 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.650+0000 7fdf35621700 1 -- start start 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.651+0000 7fdf35621700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf30069180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.651+0000 7fdf35621700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300696c0 con 0x7fdf300ff610 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.651+0000 7fdf2effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf30069180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.651+0000 7fdf2effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf30069180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44512/0 (socket says 172.21.15.196:44512) 2024-03-26T22:03:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.651+0000 7fdf2effd700 1 -- 172.21.15.196:0/856628424 learned_addr learned my addr 172.21.15.196:0/856628424 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:54.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.651+0000 7fdf2effd700 1 -- 172.21.15.196:0/856628424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf28009d00 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.652+0000 7fdf2effd700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf30069180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdf280062a0 tx=0x7fdf2800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.652+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2800e040 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.652+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf301988e0 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.652+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf28003d70 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.652+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2801c890 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.652+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf30198d80 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.653+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3018e440 con 0x7fdf300ff610 2024-03-26T22:03:54.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.653+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fdf2801cb50 con 0x7fdf300ff610 2024-03-26T22:03:54.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.653+0000 7fdf27fff700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdf1c0387c0 0x7fdf1c03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:54.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.653+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdf2804c9a0 con 0x7fdf300ff610 2024-03-26T22:03:54.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.653+0000 7fdf2e7fc700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdf1c0387c0 0x7fdf1c03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:54.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.654+0000 7fdf2e7fc700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdf1c0387c0 0x7fdf1c03ac80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf18006fd0 tx=0x7fdf18006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:54.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.658+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdf2801a3b0 con 0x7fdf300ff610 2024-03-26T22:03:54.925 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.925+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf300623e0 con 0x7fdf300ff610 2024-03-26T22:03:54.925 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.926+0000 7fdf27fff700 1 -- 172.21.15.196:0/856628424 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fdf300623e0 con 0x7fdf300ff610 2024-03-26T22:03:54.926 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:54.926 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:54.927 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdf1c0387c0 msgr2=0x7fdf1c03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdf1c0387c0 0x7fdf1c03ac80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf18006fd0 tx=0x7fdf18006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 msgr2=0x7fdf30069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf30069180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdf280062a0 tx=0x7fdf2800bd40 comp rx=0 tx=0).stop 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 shutdown_connections 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fdf1c0387c0 0x7fdf1c03ac80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 --2- 172.21.15.196:0/856628424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf300ff610 0x7fdf30069180 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 >> 172.21.15.196:0/856628424 conn(0x7fdf300fabb0 msgr2=0x7fdf300fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 shutdown_connections 2024-03-26T22:03:54.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:54.928+0000 7fdf35621700 1 -- 172.21.15.196:0/856628424 wait complete. 2024-03-26T22:03:54.929 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:55.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:55 smithi194 ceph-mon[29693]: pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:55.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:55 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/856628424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:03:56.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:03:56.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:03:56.798 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:56.843 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:03:58.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:57 smithi194 ceph-mon[29693]: pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 -- 172.21.15.196:0/3808957509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 msgr2=0x7fbf101035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 --2- 172.21.15.196:0/3808957509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf101035c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009910 tx=0x7fbf00009c20 comp rx=0 tx=0).stop 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 -- 172.21.15.196:0/3808957509 shutdown_connections 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 --2- 172.21.15.196:0/3808957509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf101035c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 -- 172.21.15.196:0/3808957509 >> 172.21.15.196:0/3808957509 conn(0x7fbf100fac00 msgr2=0x7fbf100fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 -- 172.21.15.196:0/3808957509 shutdown_connections 2024-03-26T22:03:58.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.359+0000 7fbf14a11700 1 -- 172.21.15.196:0/3808957509 wait complete. 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf14a11700 1 Processor -- start 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf14a11700 1 -- start start 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf14a11700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf10195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf14a11700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10196090 con 0x7fbf101011d0 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf0e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf10195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf0e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf10195b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44518/0 (socket says 172.21.15.196:44518) 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.360+0000 7fbf0e59c700 1 -- 172.21.15.196:0/1033060630 learned_addr learned my addr 172.21.15.196:0/1033060630 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbf0e59c700 1 -- 172.21.15.196:0/1033060630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000095f0 con 0x7fbf101011d0 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbf0e59c700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf10195b50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009ed0 tx=0x7fbf00004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf0001d070 con 0x7fbf101011d0 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf10196290 con 0x7fbf101011d0 2024-03-26T22:03:58.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf10196730 con 0x7fbf101011d0 2024-03-26T22:03:58.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf0000be20 con 0x7fbf101011d0 2024-03-26T22:03:58.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.361+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf0000f460 con 0x7fbf101011d0 2024-03-26T22:03:58.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.362+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1018f570 con 0x7fbf101011d0 2024-03-26T22:03:58.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.362+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fbf0000b960 con 0x7fbf101011d0 2024-03-26T22:03:58.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.362+0000 7fbeff7fe700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbef80387d0 0x7fbef803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:03:58.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.362+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbf0004c870 con 0x7fbf101011d0 2024-03-26T22:03:58.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.362+0000 7fbf0dd9b700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbef80387d0 0x7fbef803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:03:58.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.363+0000 7fbf0dd9b700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbef80387d0 0x7fbef803ac90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbf04006fd0 tx=0x7fbf04006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:03:58.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.367+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbf000227e0 con 0x7fbf101011d0 2024-03-26T22:03:58.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.636+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf100623e0 con 0x7fbf101011d0 2024-03-26T22:03:58.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.637+0000 7fbeff7fe700 1 -- 172.21.15.196:0/1033060630 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbf100623e0 con 0x7fbf101011d0 2024-03-26T22:03:58.636 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:03:58.636 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.638+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbef80387d0 msgr2=0x7fbef803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.638+0000 7fbf14a11700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbef80387d0 0x7fbef803ac90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbf04006fd0 tx=0x7fbf04006e40 comp rx=0 tx=0).stop 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.638+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 msgr2=0x7fbf10195b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.638+0000 7fbf14a11700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf10195b50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009ed0 tx=0x7fbf00004690 comp rx=0 tx=0).stop 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.639+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 shutdown_connections 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.639+0000 7fbf14a11700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbef80387d0 0x7fbef803ac90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:58.638 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.639+0000 7fbf14a11700 1 --2- 172.21.15.196:0/1033060630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf101011d0 0x7fbf10195b50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:03:58.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.639+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 >> 172.21.15.196:0/1033060630 conn(0x7fbf100fac00 msgr2=0x7fbf100fb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:03:58.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.639+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 shutdown_connections 2024-03-26T22:03:58.639 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:03:58.639+0000 7fbf14a11700 1 -- 172.21.15.196:0/1033060630 wait complete. 2024-03-26T22:03:58.639 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:03:59.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:58 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1033060630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:00.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:03:59 smithi194 ceph-mon[29693]: pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:00.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:00.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:00.521 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:00.567 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:02.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:01 smithi194 ceph-mon[29693]: pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:02.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 -- 172.21.15.196:0/3100825004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 msgr2=0x7f03cc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:02.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 --2- 172.21.15.196:0/3100825004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0ffa30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009910 tx=0x7f03bc009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:02.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 -- 172.21.15.196:0/3100825004 shutdown_connections 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 --2- 172.21.15.196:0/3100825004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0ffa30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 -- 172.21.15.196:0/3100825004 >> 172.21.15.196:0/3100825004 conn(0x7f03cc0fabb0 msgr2=0x7f03cc0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 -- 172.21.15.196:0/3100825004 shutdown_connections 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.217+0000 7f03d4527700 1 -- 172.21.15.196:0/3100825004 wait complete. 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d4527700 1 Processor -- start 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d4527700 1 -- start start 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d4527700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d4527700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0fcf10 con 0x7f03cc0ff610 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d22c3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d22c3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0fc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:44538/0 (socket says 172.21.15.196:44538) 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.218+0000 7f03d22c3700 1 -- 172.21.15.196:0/189300715 learned_addr learned my addr 172.21.15.196:0/189300715 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:02.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03d22c3700 1 -- 172.21.15.196:0/189300715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc0095f0 con 0x7f03cc0ff610 2024-03-26T22:04:02.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03d22c3700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0fc9d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006010 tx=0x7f03bc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:02.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03bc01d070 con 0x7f03cc0ff610 2024-03-26T22:04:02.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03bc005dc0 con 0x7f03cc0ff610 2024-03-26T22:04:02.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03bc022e90 con 0x7f03cc0ff610 2024-03-26T22:04:02.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc0fb1a0 con 0x7f03cc0ff610 2024-03-26T22:04:02.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.219+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc0fb640 con 0x7f03cc0ff610 2024-03-26T22:04:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.220+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f03bc022470 con 0x7f03cc0ff610 2024-03-26T22:04:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.220+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc0fc050 con 0x7f03cc0ff610 2024-03-26T22:04:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.220+0000 7f03c37fe700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f03b8038760 0x7f03b803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.220+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f03bc04c9e0 con 0x7f03cc0ff610 2024-03-26T22:04:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.220+0000 7f03d1ac2700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f03b8038760 0x7f03b803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.221+0000 7f03d1ac2700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f03b8038760 0x7f03b803ac20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03c8006fd0 tx=0x7f03c8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:02.225 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.226+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f03bc00b960 con 0x7f03cc0ff610 2024-03-26T22:04:02.506 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.507+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03cc062430 con 0x7f03cc0ff610 2024-03-26T22:04:02.507 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.507+0000 7f03c37fe700 1 -- 172.21.15.196:0/189300715 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f03bc027020 con 0x7f03cc0ff610 2024-03-26T22:04:02.507 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:02.507 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f03b8038760 msgr2=0x7f03b803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f03b8038760 0x7f03b803ac20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03c8006fd0 tx=0x7f03c8006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 msgr2=0x7f03cc0fc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0fc9d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006010 tx=0x7f03bc009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 shutdown_connections 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f03b8038760 0x7f03b803ac20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 --2- 172.21.15.196:0/189300715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03cc0ff610 0x7f03cc0fc9d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:02.509 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 >> 172.21.15.196:0/189300715 conn(0x7f03cc0fabb0 msgr2=0x7f03cc068bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:02.510 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.509+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 shutdown_connections 2024-03-26T22:04:02.510 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:02.510+0000 7f03d4527700 1 -- 172.21.15.196:0/189300715 wait complete. 2024-03-26T22:04:02.510 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:03.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:02 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/189300715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:04.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:03 smithi194 ceph-mon[29693]: pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:04.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:04.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:04.376 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:04.422 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:05.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:05 smithi194 ceph-mon[29693]: pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:06.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 -- 172.21.15.196:0/1000945035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 msgr2=0x7f5250100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 --2- 172.21.15.196:0/1000945035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250100b40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5240009910 tx=0x7f5240009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 -- 172.21.15.196:0/1000945035 shutdown_connections 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 --2- 172.21.15.196:0/1000945035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250100b40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 -- 172.21.15.196:0/1000945035 >> 172.21.15.196:0/1000945035 conn(0x7f52500fbcc0 msgr2=0x7f52500fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 -- 172.21.15.196:0/1000945035 shutdown_connections 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.018+0000 7f5256384700 1 -- 172.21.15.196:0/1000945035 wait complete. 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.019+0000 7f5256384700 1 Processor -- start 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.019+0000 7f5256384700 1 -- start start 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.019+0000 7f5256384700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:06.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.019+0000 7f5256384700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250074f60 con 0x7f5250100720 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:59888/0 (socket says 172.21.15.196:59888) 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524ffff700 1 -- 172.21.15.196:0/2218012500 learned_addr learned my addr 172.21.15.196:0/2218012500 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524ffff700 1 -- 172.21.15.196:0/2218012500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52400095f0 con 0x7f5250100720 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524ffff700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250074a20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f524001d070 con 0x7f5250100720 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52500731a0 con 0x7f5250100720 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5240005dc0 con 0x7f5250100720 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5240022e90 con 0x7f5250100720 2024-03-26T22:04:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.020+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5250073640 con 0x7f5250100720 2024-03-26T22:04:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.021+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5250106e80 con 0x7f5250100720 2024-03-26T22:04:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.021+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f5240022470 con 0x7f5250100720 2024-03-26T22:04:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.021+0000 7f524d7fa700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5238038760 0x7f523803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.021+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f524004c8b0 con 0x7f5250100720 2024-03-26T22:04:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.022+0000 7f524f7fe700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5238038760 0x7f523803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.022+0000 7f524f7fe700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5238038760 0x7f523803ac20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5244006fd0 tx=0x7f5244006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:06.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.027+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f524000b960 con 0x7f5250100720 2024-03-26T22:04:06.292 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.293+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52500623e0 con 0x7f5250100720 2024-03-26T22:04:06.293 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.293+0000 7f524d7fa700 1 -- 172.21.15.196:0/2218012500 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5240027030 con 0x7f5250100720 2024-03-26T22:04:06.293 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:06.293 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:06.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5238038760 msgr2=0x7f523803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:06.295 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5238038760 0x7f523803ac20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5244006fd0 tx=0x7f5244006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 msgr2=0x7f5250074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250074a20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 shutdown_connections 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f5238038760 0x7f523803ac20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 --2- 172.21.15.196:0/2218012500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5250100720 0x7f5250074a20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 >> 172.21.15.196:0/2218012500 conn(0x7f52500fbcc0 msgr2=0x7f52500fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 shutdown_connections 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:06.296+0000 7f5256384700 1 -- 172.21.15.196:0/2218012500 wait complete. 2024-03-26T22:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:06.665 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:06 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2218012500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:07.665 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:07 smithi194 ceph-mon[29693]: pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:07.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:07.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:08.130 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:08.176 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:09.659 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.659+0000 7f3e1b61a700 1 -- 172.21.15.196:0/1878082894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 msgr2=0x7f3e14100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.659+0000 7f3e1b61a700 1 --2- 172.21.15.196:0/1878082894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14100b40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009910 tx=0x7f3e10009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.659+0000 7f3e1b61a700 1 -- 172.21.15.196:0/1878082894 shutdown_connections 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.659+0000 7f3e1b61a700 1 --2- 172.21.15.196:0/1878082894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14100b40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.659+0000 7f3e1b61a700 1 -- 172.21.15.196:0/1878082894 >> 172.21.15.196:0/1878082894 conn(0x7f3e140fbcc0 msgr2=0x7f3e140fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.659+0000 7f3e1b61a700 1 -- 172.21.15.196:0/1878082894 shutdown_connections 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.660+0000 7f3e1b61a700 1 -- 172.21.15.196:0/1878082894 wait complete. 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.660+0000 7f3e1b61a700 1 Processor -- start 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.660+0000 7f3e1b61a700 1 -- start start 2024-03-26T22:04:09.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.660+0000 7f3e1b61a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:09.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.660+0000 7f3e1b61a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14074f60 con 0x7f3e14100720 2024-03-26T22:04:09.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.661+0000 7f3e193b6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:09.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.661+0000 7f3e193b6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:59904/0 (socket says 172.21.15.196:59904) 2024-03-26T22:04:09.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.661+0000 7f3e193b6700 1 -- 172.21.15.196:0/873985584 learned_addr learned my addr 172.21.15.196:0/873985584 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.661+0000 7f3e193b6700 1 -- 172.21.15.196:0/873985584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e100095f0 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.661+0000 7f3e193b6700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14074a20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3e10006010 tx=0x7f3e10009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.661+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1001d070 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.662+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e140731a0 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.662+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e10005dc0 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.662+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e10022e90 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.662+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e14073640 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.662+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14106e80 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.662+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3e10022470 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.663+0000 7f3e0a7fc700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3e00038760 0x7f3e0003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.663+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e1004c870 con 0x7f3e14100720 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.663+0000 7f3e18bb5700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3e00038760 0x7f3e0003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:09.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.663+0000 7f3e18bb5700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3e00038760 0x7f3e0003ac20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3e04006fd0 tx=0x7f3e04006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:09.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.668+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3e1000f4f0 con 0x7f3e14100720 2024-03-26T22:04:09.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.935+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e140623e0 con 0x7f3e14100720 2024-03-26T22:04:09.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.936+0000 7f3e0a7fc700 1 -- 172.21.15.196:0/873985584 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3e10027070 con 0x7f3e14100720 2024-03-26T22:04:09.936 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:09.936 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:09.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3e00038760 msgr2=0x7f3e0003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:09.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3e00038760 0x7f3e0003ac20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3e04006fd0 tx=0x7f3e04006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 msgr2=0x7f3e14074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14074a20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3e10006010 tx=0x7f3e10009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 shutdown_connections 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3e00038760 0x7f3e0003ac20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 --2- 172.21.15.196:0/873985584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e14100720 0x7f3e14074a20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 >> 172.21.15.196:0/873985584 conn(0x7f3e140fbcc0 msgr2=0x7f3e140fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 shutdown_connections 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:09.938+0000 7f3e1b61a700 1 -- 172.21.15.196:0/873985584 wait complete. 2024-03-26T22:04:09.939 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:10.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:09 smithi194 ceph-mon[29693]: pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:10.914 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:10 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/873985584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:11.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:11.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:11.822 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:11.868 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:12.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:11 smithi194 ceph-mon[29693]: pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 -- 172.21.15.196:0/749931877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 msgr2=0x7f25901035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 --2- 172.21.15.196:0/749931877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f25901035a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f258c009910 tx=0x7f258c009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 -- 172.21.15.196:0/749931877 shutdown_connections 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 --2- 172.21.15.196:0/749931877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f25901035a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 -- 172.21.15.196:0/749931877 >> 172.21.15.196:0/749931877 conn(0x7f25900fab80 msgr2=0x7f25900fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 -- 172.21.15.196:0/749931877 shutdown_connections 2024-03-26T22:04:13.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.269+0000 7f259811d700 1 -- 172.21.15.196:0/749931877 wait complete. 2024-03-26T22:04:13.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f259811d700 1 Processor -- start 2024-03-26T22:04:13.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f259811d700 1 -- start start 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f259811d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f2590195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f259811d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25901960a0 con 0x7f25901011b0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f2595eb9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f2590195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f2595eb9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f2590195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34342/0 (socket says 172.21.15.196:34342) 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f2595eb9700 1 -- 172.21.15.196:0/4222890279 learned_addr learned my addr 172.21.15.196:0/4222890279 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.270+0000 7f2595eb9700 1 -- 172.21.15.196:0/4222890279 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c0095f0 con 0x7f25901011b0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.271+0000 7f2595eb9700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f2590195b60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f258c009ed0 tx=0x7f258c004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.271+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f258c01d070 con 0x7f25901011b0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.271+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25901962a0 con 0x7f25901011b0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.271+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f258c00be20 con 0x7f25901011b0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.271+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f258c00f460 con 0x7f25901011b0 2024-03-26T22:04:13.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.271+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2590196740 con 0x7f25901011b0 2024-03-26T22:04:13.272 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.272+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f258c00f650 con 0x7f25901011b0 2024-03-26T22:04:13.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.272+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259018f550 con 0x7f25901011b0 2024-03-26T22:04:13.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.272+0000 7f2586ffd700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f257c0387d0 0x7f257c03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:13.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.272+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f258c04caa0 con 0x7f25901011b0 2024-03-26T22:04:13.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.272+0000 7f25956b8700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f257c0387d0 0x7f257c03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:13.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.273+0000 7f25956b8700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f257c0387d0 0x7f257c03ac90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2580006fd0 tx=0x7f2580006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:13.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.277+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f258c02a360 con 0x7f25901011b0 2024-03-26T22:04:13.547 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.547+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25900623e0 con 0x7f25901011b0 2024-03-26T22:04:13.547 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.548+0000 7f2586ffd700 1 -- 172.21.15.196:0/4222890279 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f25900623e0 con 0x7f25901011b0 2024-03-26T22:04:13.547 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:13.548 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:13.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f257c0387d0 msgr2=0x7f257c03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:13.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f257c0387d0 0x7f257c03ac90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2580006fd0 tx=0x7f2580006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:13.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 msgr2=0x7f2590195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:13.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f2590195b60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f258c009ed0 tx=0x7f258c004690 comp rx=0 tx=0).stop 2024-03-26T22:04:13.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 shutdown_connections 2024-03-26T22:04:13.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f257c0387d0 0x7f257c03ac90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:13.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 --2- 172.21.15.196:0/4222890279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25901011b0 0x7f2590195b60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:13.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 >> 172.21.15.196:0/4222890279 conn(0x7f25900fab80 msgr2=0x7f25900fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:13.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 shutdown_connections 2024-03-26T22:04:13.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:13.550+0000 7f259811d700 1 -- 172.21.15.196:0/4222890279 wait complete. 2024-03-26T22:04:13.550 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:14.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:13 smithi194 ceph-mon[29693]: pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:14.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:13 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4222890279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:15.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:15.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:15.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:15 smithi194 ceph-mon[29693]: pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:15.431 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:15.478 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 -- 172.21.15.196:0/2316636902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 msgr2=0x7efbf00ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 --2- 172.21.15.196:0/2316636902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf00ffa40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efbec009910 tx=0x7efbec009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 -- 172.21.15.196:0/2316636902 shutdown_connections 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 --2- 172.21.15.196:0/2316636902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf00ffa40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 -- 172.21.15.196:0/2316636902 >> 172.21.15.196:0/2316636902 conn(0x7efbf00fab80 msgr2=0x7efbf00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 -- 172.21.15.196:0/2316636902 shutdown_connections 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.029+0000 7efbf8493700 1 -- 172.21.15.196:0/2316636902 wait complete. 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf8493700 1 Processor -- start 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf8493700 1 -- start start 2024-03-26T22:04:17.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf8493700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf0195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf8493700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf0196130 con 0x7efbf00ff620 2024-03-26T22:04:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf622f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf0195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf622f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf0195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34364/0 (socket says 172.21.15.196:34364) 2024-03-26T22:04:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf622f700 1 -- 172.21.15.196:0/881614598 learned_addr learned my addr 172.21.15.196:0/881614598 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.030+0000 7efbf622f700 1 -- 172.21.15.196:0/881614598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec0095f0 con 0x7efbf00ff620 2024-03-26T22:04:17.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.031+0000 7efbf622f700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf0195bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efbec006010 tx=0x7efbec009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:17.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.031+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbec01d070 con 0x7efbf00ff620 2024-03-26T22:04:17.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.031+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0196330 con 0x7efbf00ff620 2024-03-26T22:04:17.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.031+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbec005dc0 con 0x7efbf00ff620 2024-03-26T22:04:17.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.031+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbec022e90 con 0x7efbf00ff620 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.031+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf01967d0 con 0x7efbf00ff620 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.032+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf018f720 con 0x7efbf00ff620 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.032+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7efbec022470 con 0x7efbf00ff620 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.032+0000 7efbe77fe700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbdc038760 0x7efbdc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.032+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efbec04c7d0 con 0x7efbf00ff620 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.032+0000 7efbf5a2e700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbdc038760 0x7efbdc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:17.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.033+0000 7efbf5a2e700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbdc038760 0x7efbdc03ac20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efbe0006fd0 tx=0x7efbe0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:17.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.038+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efbec00b960 con 0x7efbf00ff620 2024-03-26T22:04:17.312 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.313+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efbf00623e0 con 0x7efbf00ff620 2024-03-26T22:04:17.313 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.313+0000 7efbe77fe700 1 -- 172.21.15.196:0/881614598 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7efbf00623e0 con 0x7efbf00ff620 2024-03-26T22:04:17.313 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:17.313 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:17.315 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbdc038760 msgr2=0x7efbdc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:17.315 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbdc038760 0x7efbdc03ac20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efbe0006fd0 tx=0x7efbe0006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:17.315 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 msgr2=0x7efbf0195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:17.315 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf0195bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efbec006010 tx=0x7efbec009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:17.315 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 shutdown_connections 2024-03-26T22:04:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7efbdc038760 0x7efbdc03ac20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 --2- 172.21.15.196:0/881614598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efbf00ff620 0x7efbf0195bf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 >> 172.21.15.196:0/881614598 conn(0x7efbf00fab80 msgr2=0x7efbf00682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 shutdown_connections 2024-03-26T22:04:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:17.316+0000 7efbf8493700 1 -- 172.21.15.196:0/881614598 wait complete. 2024-03-26T22:04:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:18.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:17 smithi194 ceph-mon[29693]: pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:18.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:17 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/881614598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:18.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:18.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:19.201 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:19.247 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:20.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:19 smithi194 ceph-mon[29693]: pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:20.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- 172.21.15.196:0/1553365061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 msgr2=0x7fe55c0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:20.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 --2- 172.21.15.196:0/1553365061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c0ffa60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009910 tx=0x7fe54c009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:20.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- 172.21.15.196:0/1553365061 shutdown_connections 2024-03-26T22:04:20.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 --2- 172.21.15.196:0/1553365061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c0ffa60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:20.674 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- 172.21.15.196:0/1553365061 >> 172.21.15.196:0/1553365061 conn(0x7fe55c0fac00 msgr2=0x7fe55c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:20.674 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- 172.21.15.196:0/1553365061 shutdown_connections 2024-03-26T22:04:20.674 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- 172.21.15.196:0/1553365061 wait complete. 2024-03-26T22:04:20.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 Processor -- start 2024-03-26T22:04:20.675 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- start start 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.674+0000 7fe561c25700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c1960e0 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe55b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe55b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34392/0 (socket says 172.21.15.196:34392) 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe55b7fe700 1 -- 172.21.15.196:0/1819690072 learned_addr learned my addr 172.21.15.196:0/1819690072 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe55b7fe700 1 -- 172.21.15.196:0/1819690072 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c0095f0 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe55b7fe700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c195ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe54c006010 tx=0x7fe54c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe54c01d070 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe54c005dc0 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55c1962e0 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe54c022e90 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.675+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55c196780 con 0x7fe55c0ff640 2024-03-26T22:04:20.676 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.676+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55c0623e0 con 0x7fe55c0ff640 2024-03-26T22:04:20.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.676+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe54c022470 con 0x7fe55c0ff640 2024-03-26T22:04:20.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.676+0000 7fe5597fa700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe548038760 0x7fe54803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:20.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.677+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe54c04c7f0 con 0x7fe55c0ff640 2024-03-26T22:04:20.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.677+0000 7fe553fff700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe548038760 0x7fe54803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:20.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.677+0000 7fe553fff700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe548038760 0x7fe54803ac20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe544006fd0 tx=0x7fe544006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:20.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.682+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe54c00b960 con 0x7fe55c0ff640 2024-03-26T22:04:20.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.948+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe55c1056a0 con 0x7fe55c0ff640 2024-03-26T22:04:20.948 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.949+0000 7fe5597fa700 1 -- 172.21.15.196:0/1819690072 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe54c027030 con 0x7fe55c0ff640 2024-03-26T22:04:20.949 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:20.949 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe548038760 msgr2=0x7fe54803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe548038760 0x7fe54803ac20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe544006fd0 tx=0x7fe544006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 msgr2=0x7fe55c195ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c195ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe54c006010 tx=0x7fe54c009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 shutdown_connections 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe548038760 0x7fe54803ac20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 --2- 172.21.15.196:0/1819690072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe55c0ff640 0x7fe55c195ba0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 >> 172.21.15.196:0/1819690072 conn(0x7fe55c0fac00 msgr2=0x7fe55c0fb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 shutdown_connections 2024-03-26T22:04:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:20.951+0000 7fe561c25700 1 -- 172.21.15.196:0/1819690072 wait complete. 2024-03-26T22:04:20.952 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:22.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:21 smithi194 ceph-mon[29693]: pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:22.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:21 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1819690072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:22.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:22.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:22.728 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:22.773 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:24.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:23 smithi194 ceph-mon[29693]: pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 -- 172.21.15.196:0/2138014750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 msgr2=0x7fbee0100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 --2- 172.21.15.196:0/2138014750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0100b30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009910 tx=0x7fbedc009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 -- 172.21.15.196:0/2138014750 shutdown_connections 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 --2- 172.21.15.196:0/2138014750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0100b30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 -- 172.21.15.196:0/2138014750 >> 172.21.15.196:0/2138014750 conn(0x7fbee00fbcd0 msgr2=0x7fbee00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 -- 172.21.15.196:0/2138014750 shutdown_connections 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.210+0000 7fbee7629700 1 -- 172.21.15.196:0/2138014750 wait complete. 2024-03-26T22:04:24.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee7629700 1 Processor -- start 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee7629700 1 -- start start 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee7629700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee7629700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0074f60 con 0x7fbee0100710 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee53c5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee53c5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49796/0 (socket says 172.21.15.196:49796) 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee53c5700 1 -- 172.21.15.196:0/2244868149 learned_addr learned my addr 172.21.15.196:0/2244868149 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.211+0000 7fbee53c5700 1 -- 172.21.15.196:0/2244868149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc0095f0 con 0x7fbee0100710 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.212+0000 7fbee53c5700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0074a20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006010 tx=0x7fbedc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.212+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc01d070 con 0x7fbee0100710 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.212+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee00731a0 con 0x7fbee0100710 2024-03-26T22:04:24.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.212+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbedc005dc0 con 0x7fbee0100710 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.212+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc022e90 con 0x7fbee0100710 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.212+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee0073640 con 0x7fbee0100710 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.213+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee0106e70 con 0x7fbee0100710 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.213+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fbedc022470 con 0x7fbee0100710 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.213+0000 7fbed67fc700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbecc038760 0x7fbecc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.213+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbedc04c7f0 con 0x7fbee0100710 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.213+0000 7fbee4bc4700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbecc038760 0x7fbecc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:24.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.213+0000 7fbee4bc4700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbecc038760 0x7fbecc03ac20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbed0006fd0 tx=0x7fbed0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:24.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.218+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbedc00b960 con 0x7fbee0100710 2024-03-26T22:04:24.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.485+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbee00623e0 con 0x7fbee0100710 2024-03-26T22:04:24.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.486+0000 7fbed67fc700 1 -- 172.21.15.196:0/2244868149 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbedc027030 con 0x7fbee0100710 2024-03-26T22:04:24.486 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:24.486 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:24.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.488+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbecc038760 msgr2=0x7fbecc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:24.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.488+0000 7fbee7629700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbecc038760 0x7fbecc03ac20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbed0006fd0 tx=0x7fbed0006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:24.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.488+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 msgr2=0x7fbee0074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:24.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.488+0000 7fbee7629700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0074a20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006010 tx=0x7fbedc009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.489+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 shutdown_connections 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.489+0000 7fbee7629700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbecc038760 0x7fbecc03ac20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.489+0000 7fbee7629700 1 --2- 172.21.15.196:0/2244868149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee0100710 0x7fbee0074a20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.489+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 >> 172.21.15.196:0/2244868149 conn(0x7fbee00fbcd0 msgr2=0x7fbee00fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.489+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 shutdown_connections 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:24.489+0000 7fbee7629700 1 -- 172.21.15.196:0/2244868149 wait complete. 2024-03-26T22:04:24.489 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:25.414 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:25 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2244868149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:25.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:25 smithi194 ceph-mon[29693]: pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:26.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:26.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:26.302 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:26.344 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/330291649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 msgr2=0x7f4bcc0ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 --2- 172.21.15.196:0/330291649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc0ff200 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009910 tx=0x7f4bbc009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/330291649 shutdown_connections 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 --2- 172.21.15.196:0/330291649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc0ff200 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/330291649 >> 172.21.15.196:0/330291649 conn(0x7f4bcc0fa380 msgr2=0x7f4bcc0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/330291649 shutdown_connections 2024-03-26T22:04:27.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.938+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/330291649 wait complete. 2024-03-26T22:04:27.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.939+0000 7f4bd0f2a700 1 Processor -- start 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.939+0000 7f4bd0f2a700 1 -- start start 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.939+0000 7f4bd0f2a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc18bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.939+0000 7f4bd0f2a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc18c3e0 con 0x7f4bcc0fede0 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.939+0000 7f4bca59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc18bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bca59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc18bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49810/0 (socket says 172.21.15.196:49810) 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bca59c700 1 -- 172.21.15.196:0/1252272294 learned_addr learned my addr 172.21.15.196:0/1252272294 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bca59c700 1 -- 172.21.15.196:0/1252272294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0095f0 con 0x7f4bcc0fede0 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bca59c700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc18bea0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc006010 tx=0x7f4bbc0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bbc01d070 con 0x7f4bcc0fede0 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc18a620 con 0x7f4bcc0fede0 2024-03-26T22:04:27.941 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bbc00be20 con 0x7f4bcc0fede0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bbc022de0 con 0x7f4bcc0fede0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.940+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc18aac0 con 0x7f4bcc0fede0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.941+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f4bbc00b960 con 0x7f4bcc0fede0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.941+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc105540 con 0x7f4bcc0fede0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.941+0000 7f4bbb7fe700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4bb40387b0 0x7f4bb403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.941+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4bbc04c980 con 0x7f4bcc0fede0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.941+0000 7f4bc9d9b700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4bb40387b0 0x7f4bb403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:27.942 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.942+0000 7f4bc9d9b700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4bb40387b0 0x7f4bb403ac70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0006fd0 tx=0x7f4bc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:27.945 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:27.946+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4bbc0226c0 con 0x7f4bcc0fede0 2024-03-26T22:04:28.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:27 smithi194 ceph-mon[29693]: pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:28.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.216+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bcc0623e0 con 0x7f4bcc0fede0 2024-03-26T22:04:28.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.217+0000 7f4bbb7fe700 1 -- 172.21.15.196:0/1252272294 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4bbc027030 con 0x7f4bcc0fede0 2024-03-26T22:04:28.216 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:28.216 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:28.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4bb40387b0 msgr2=0x7f4bb403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:28.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4bb40387b0 0x7f4bb403ac70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0006fd0 tx=0x7f4bc0006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:28.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 msgr2=0x7f4bcc18bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc18bea0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc006010 tx=0x7f4bbc0050b0 comp rx=0 tx=0).stop 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 shutdown_connections 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f4bb40387b0 0x7f4bb403ac70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 --2- 172.21.15.196:0/1252272294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4bcc0fede0 0x7f4bcc18bea0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 >> 172.21.15.196:0/1252272294 conn(0x7f4bcc0fa380 msgr2=0x7f4bcc18c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 shutdown_connections 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:28.219+0000 7f4bd0f2a700 1 -- 172.21.15.196:0/1252272294 wait complete. 2024-03-26T22:04:28.219 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:29.164 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:28 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1252272294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:29.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:29.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:30.001 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:30.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:29 smithi194 ceph-mon[29693]: pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 -- 172.21.15.196:0/1619515570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 msgr2=0x7f2790103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 --2- 172.21.15.196:0/1619515570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790103640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f278800b0a0 tx=0x7f278800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 -- 172.21.15.196:0/1619515570 shutdown_connections 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 --2- 172.21.15.196:0/1619515570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790103640 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 -- 172.21.15.196:0/1619515570 >> 172.21.15.196:0/1619515570 conn(0x7f27900fab80 msgr2=0x7f27900fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 -- 172.21.15.196:0/1619515570 shutdown_connections 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.592+0000 7f27954e0700 1 -- 172.21.15.196:0/1619515570 wait complete. 2024-03-26T22:04:31.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.593+0000 7f27954e0700 1 Processor -- start 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.593+0000 7f27954e0700 1 -- start start 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.593+0000 7f27954e0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.593+0000 7f27954e0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901960a0 con 0x7f2790101250 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.593+0000 7f278effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.593+0000 7f278effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:49824/0 (socket says 172.21.15.196:49824) 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f278effd700 1 -- 172.21.15.196:0/2541302088 learned_addr learned my addr 172.21.15.196:0/2541302088 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f278effd700 1 -- 172.21.15.196:0/2541302088 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788009d00 con 0x7f2790101250 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f278effd700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790195b60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f27880062a0 tx=0x7f278800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f278800e040 con 0x7f2790101250 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27901962a0 con 0x7f2790101250 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2788003d70 con 0x7f2790101250 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f278801c8d0 con 0x7f2790101250 2024-03-26T22:04:31.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.594+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2790196740 con 0x7f2790101250 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.595+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279018f550 con 0x7f2790101250 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.595+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f278801cb90 con 0x7f2790101250 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.595+0000 7f2777fff700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f27780387c0 0x7f277803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.595+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f278804ca40 con 0x7f2790101250 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.595+0000 7f278e7fc700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f27780387c0 0x7f277803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.596+0000 7f278e7fc700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f27780387c0 0x7f277803ac80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2784006fd0 tx=0x7f2784006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:31.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.601+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f278801a590 con 0x7f2790101250 2024-03-26T22:04:31.868 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.869+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27900623e0 con 0x7f2790101250 2024-03-26T22:04:31.870 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.869+0000 7f2777fff700 1 -- 172.21.15.196:0/2541302088 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f278801a3b0 con 0x7f2790101250 2024-03-26T22:04:31.870 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:31.870 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:31.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f27780387c0 msgr2=0x7f277803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:31.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f27780387c0 0x7f277803ac80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2784006fd0 tx=0x7f2784006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 msgr2=0x7f2790195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790195b60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f27880062a0 tx=0x7f278800bd40 comp rx=0 tx=0).stop 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 shutdown_connections 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f27780387c0 0x7f277803ac80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.871+0000 7f27954e0700 1 --2- 172.21.15.196:0/2541302088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2790101250 0x7f2790195b60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.872+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 >> 172.21.15.196:0/2541302088 conn(0x7f27900fab80 msgr2=0x7f27900fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.872+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 shutdown_connections 2024-03-26T22:04:31.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:31.872+0000 7f27954e0700 1 -- 172.21.15.196:0/2541302088 wait complete. 2024-03-26T22:04:31.873 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:32.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:31 smithi194 ceph-mon[29693]: pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:33.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:32 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2541302088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:33.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:33.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:33.890 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:33.934 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:34.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:33 smithi194 ceph-mon[29693]: pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:35.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:35 smithi194 ceph-mon[29693]: pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.464+0000 7fa852086700 1 -- 172.21.15.196:0/3678161978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 msgr2=0x7fa84c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.464+0000 7fa852086700 1 --2- 172.21.15.196:0/3678161978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0ffa40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009910 tx=0x7fa83c009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.464+0000 7fa852086700 1 -- 172.21.15.196:0/3678161978 shutdown_connections 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.464+0000 7fa852086700 1 --2- 172.21.15.196:0/3678161978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0ffa40 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.464+0000 7fa852086700 1 -- 172.21.15.196:0/3678161978 >> 172.21.15.196:0/3678161978 conn(0x7fa84c0fabc0 msgr2=0x7fa84c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.464+0000 7fa852086700 1 -- 172.21.15.196:0/3678161978 shutdown_connections 2024-03-26T22:04:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.465+0000 7fa852086700 1 -- 172.21.15.196:0/3678161978 wait complete. 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.465+0000 7fa852086700 1 Processor -- start 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.465+0000 7fa852086700 1 -- start start 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.465+0000 7fa852086700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.465+0000 7fa852086700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c069a20 con 0x7fa84c0ff620 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa84b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa84b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43836/0 (socket says 172.21.15.196:43836) 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa84b7fe700 1 -- 172.21.15.196:0/2540416166 learned_addr learned my addr 172.21.15.196:0/2540416166 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa84b7fe700 1 -- 172.21.15.196:0/2540416166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0095f0 con 0x7fa84c0ff620 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa84b7fe700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0694e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa83c006010 tx=0x7fa83c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c01d070 con 0x7fa84c0ff620 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c196710 con 0x7fa84c0ff620 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa83c005dc0 con 0x7fa84c0ff620 2024-03-26T22:04:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c022e90 con 0x7fa84c0ff620 2024-03-26T22:04:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.466+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c19af70 con 0x7fa84c0ff620 2024-03-26T22:04:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.467+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa83c022470 con 0x7fa84c0ff620 2024-03-26T22:04:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.467+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c18e450 con 0x7fa84c0ff620 2024-03-26T22:04:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.467+0000 7fa8497fa700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa838038760 0x7fa83803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.467+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa83c04c850 con 0x7fa84c0ff620 2024-03-26T22:04:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.468+0000 7fa843fff700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa838038760 0x7fa83803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.468+0000 7fa843fff700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa838038760 0x7fa83803ac20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa834006fd0 tx=0x7fa834006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.473+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa83c00b960 con 0x7fa84c0ff620 2024-03-26T22:04:35.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.740+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa84c0623e0 con 0x7fa84c0ff620 2024-03-26T22:04:35.740 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.741+0000 7fa8497fa700 1 -- 172.21.15.196:0/2540416166 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa83c027020 con 0x7fa84c0ff620 2024-03-26T22:04:35.740 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:35.741 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa838038760 msgr2=0x7fa83803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa838038760 0x7fa83803ac20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa834006fd0 tx=0x7fa834006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 msgr2=0x7fa84c0694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0694e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa83c006010 tx=0x7fa83c009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 shutdown_connections 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa838038760 0x7fa83803ac20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 --2- 172.21.15.196:0/2540416166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa84c0ff620 0x7fa84c0694e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 >> 172.21.15.196:0/2540416166 conn(0x7fa84c0fabc0 msgr2=0x7fa84c0fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 shutdown_connections 2024-03-26T22:04:35.743 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:35.743+0000 7fa852086700 1 -- 172.21.15.196:0/2540416166 wait complete. 2024-03-26T22:04:35.744 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:36.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:36 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2540416166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:37.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:37.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:37.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:37 smithi194 ceph-mon[29693]: pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:37.582 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:37.627 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:39.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 -- 172.21.15.196:0/3471495375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 msgr2=0x7f238c0ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:39.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 --2- 172.21.15.196:0/3471495375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c0ffa80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f238400b0a0 tx=0x7f238400b3b0 comp rx=0 tx=0).stop 2024-03-26T22:04:39.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 -- 172.21.15.196:0/3471495375 shutdown_connections 2024-03-26T22:04:39.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 --2- 172.21.15.196:0/3471495375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c0ffa80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 -- 172.21.15.196:0/3471495375 >> 172.21.15.196:0/3471495375 conn(0x7f238c0fac20 msgr2=0x7f238c0fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 -- 172.21.15.196:0/3471495375 shutdown_connections 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.151+0000 7f23914e9700 1 -- 172.21.15.196:0/3471495375 wait complete. 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.152+0000 7f23914e9700 1 Processor -- start 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.152+0000 7f23914e9700 1 -- start start 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.152+0000 7f23914e9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c195c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.152+0000 7f23914e9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c196150 con 0x7f238c0ff660 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f238affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c195c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f238affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c195c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43856/0 (socket says 172.21.15.196:43856) 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f238affd700 1 -- 172.21.15.196:0/2225234247 learned_addr learned my addr 172.21.15.196:0/2225234247 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:39.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f238affd700 1 -- 172.21.15.196:0/2225234247 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384009d00 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f238affd700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c195c10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f23840062a0 tx=0x7f238400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2384020070 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2384003cd0 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c196350 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.153+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f238400e040 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.154+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c1967f0 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.154+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f2384004040 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.154+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c18f750 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.155+0000 7f2383fff700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f23780387b0 0x7f237803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.155+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f238404c0c0 con 0x7f238c0ff660 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.155+0000 7f238a7fc700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f23780387b0 0x7f237803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:39.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.155+0000 7f238a7fc700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f23780387b0 0x7f237803ac70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2374006fd0 tx=0x7f2374006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:39.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.160+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2384011dd0 con 0x7f238c0ff660 2024-03-26T22:04:39.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.427+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f238c0623e0 con 0x7f238c0ff660 2024-03-26T22:04:39.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.428+0000 7f2383fff700 1 -- 172.21.15.196:0/2225234247 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f238c0623e0 con 0x7f238c0ff660 2024-03-26T22:04:39.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:39.428 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:39.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f23780387b0 msgr2=0x7f237803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:39.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f23780387b0 0x7f237803ac70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2374006fd0 tx=0x7f2374006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 msgr2=0x7f238c195c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c195c10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f23840062a0 tx=0x7f238400bd40 comp rx=0 tx=0).stop 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 shutdown_connections 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f23780387b0 0x7f237803ac70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 --2- 172.21.15.196:0/2225234247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f238c0ff660 0x7f238c195c10 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 >> 172.21.15.196:0/2225234247 conn(0x7f238c0fac20 msgr2=0x7f238c068f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 shutdown_connections 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:39.430+0000 7f23914e9700 1 -- 172.21.15.196:0/2225234247 wait complete. 2024-03-26T22:04:39.430 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:40.118 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:39 smithi194 ceph-mon[29693]: pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:40.118 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:39 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2225234247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:40.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:40.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:41.239 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:42.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:41 smithi194 ceph-mon[29693]: pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 -- 172.21.15.196:0/502697379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 msgr2=0x7fe7d00ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 --2- 172.21.15.196:0/502697379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d00ffa50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009910 tx=0x7fe7c4009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 -- 172.21.15.196:0/502697379 shutdown_connections 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 --2- 172.21.15.196:0/502697379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d00ffa50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 -- 172.21.15.196:0/502697379 >> 172.21.15.196:0/502697379 conn(0x7fe7d00fabd0 msgr2=0x7fe7d00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 -- 172.21.15.196:0/502697379 shutdown_connections 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.794+0000 7fe7d6cec700 1 -- 172.21.15.196:0/502697379 wait complete. 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d6cec700 1 Processor -- start 2024-03-26T22:04:42.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d6cec700 1 -- start start 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d6cec700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d0194b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d6cec700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d0195070 con 0x7fe7d00ff630 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d4a88700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d0194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d4a88700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d0194b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43200/0 (socket says 172.21.15.196:43200) 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d4a88700 1 -- 172.21.15.196:0/3581068118 learned_addr learned my addr 172.21.15.196:0/3581068118 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.795+0000 7fe7d4a88700 1 -- 172.21.15.196:0/3581068118 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40095f0 con 0x7fe7d00ff630 2024-03-26T22:04:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.796+0000 7fe7d4a88700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d0194b30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4004f40 tx=0x7fe7c4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:42.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.796+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c401d070 con 0x7fe7d00ff630 2024-03-26T22:04:42.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.796+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7c4005dc0 con 0x7fe7d00ff630 2024-03-26T22:04:42.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.796+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d0195270 con 0x7fe7d00ff630 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.796+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c4022e40 con 0x7fe7d00ff630 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.796+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d0195710 con 0x7fe7d00ff630 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.797+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d018e460 con 0x7fe7d00ff630 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.797+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe7c40226c0 con 0x7fe7d00ff630 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.797+0000 7fe7cdffb700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe7c0038760 0x7fe7c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.797+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe7c404c8c0 con 0x7fe7d00ff630 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.797+0000 7fe7cffff700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe7c0038760 0x7fe7c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:42.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.798+0000 7fe7cffff700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe7c0038760 0x7fe7c003ac20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc006fd0 tx=0x7fe7bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:42.802+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe7c400f4b0 con 0x7fe7d00ff630 2024-03-26T22:04:43.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.071+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7d00623e0 con 0x7fe7d00ff630 2024-03-26T22:04:43.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.072+0000 7fe7cdffb700 1 -- 172.21.15.196:0/3581068118 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe7c4027070 con 0x7fe7d00ff630 2024-03-26T22:04:43.072 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:43.072 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:43.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe7c0038760 msgr2=0x7fe7c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:43.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe7c0038760 0x7fe7c003ac20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc006fd0 tx=0x7fe7bc006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:43.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 msgr2=0x7fe7d0194b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:43.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d0194b30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4004f40 tx=0x7fe7c4004740 comp rx=0 tx=0).stop 2024-03-26T22:04:43.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 shutdown_connections 2024-03-26T22:04:43.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe7c0038760 0x7fe7c003ac20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:43.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 --2- 172.21.15.196:0/3581068118 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7d00ff630 0x7fe7d0194b30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:43.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.074+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 >> 172.21.15.196:0/3581068118 conn(0x7fe7d00fabd0 msgr2=0x7fe7d00fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:43.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.075+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 shutdown_connections 2024-03-26T22:04:43.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:43.075+0000 7fe7d6cec700 1 -- 172.21.15.196:0/3581068118 wait complete. 2024-03-26T22:04:43.075 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:44.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:43 smithi194 ceph-mon[29693]: pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:44.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:43 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3581068118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:44.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:44.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:44.826 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:44.872 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:45.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:45 smithi194 ceph-mon[29693]: pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:46.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 -- 172.21.15.196:0/205658828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 msgr2=0x7f632c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:46.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 --2- 172.21.15.196:0/205658828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c0ffa40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f631c009910 tx=0x7f631c009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:46.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 -- 172.21.15.196:0/205658828 shutdown_connections 2024-03-26T22:04:46.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 --2- 172.21.15.196:0/205658828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c0ffa40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:46.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 -- 172.21.15.196:0/205658828 >> 172.21.15.196:0/205658828 conn(0x7f632c0fab80 msgr2=0x7f632c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:46.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 -- 172.21.15.196:0/205658828 shutdown_connections 2024-03-26T22:04:46.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.402+0000 7f6333c22700 1 -- 172.21.15.196:0/205658828 wait complete. 2024-03-26T22:04:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.403+0000 7f6333c22700 1 Processor -- start 2024-03-26T22:04:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.403+0000 7f6333c22700 1 -- start start 2024-03-26T22:04:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.403+0000 7f6333c22700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c194ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.403+0000 7f6333c22700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c195000 con 0x7f632c0ff620 2024-03-26T22:04:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.403+0000 7f63319be700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c194ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.403+0000 7f63319be700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c194ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43216/0 (socket says 172.21.15.196:43216) 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f63319be700 1 -- 172.21.15.196:0/2513251091 learned_addr learned my addr 172.21.15.196:0/2513251091 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f63319be700 1 -- 172.21.15.196:0/2513251091 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c0095f0 con 0x7f632c0ff620 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f63319be700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c194ac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f631c006010 tx=0x7f631c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f631c01d070 con 0x7f632c0ff620 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c195200 con 0x7f632c0ff620 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f631c005dc0 con 0x7f632c0ff620 2024-03-26T22:04:46.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c1956a0 con 0x7f632c0ff620 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.404+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f631c022e90 con 0x7f632c0ff620 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.405+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f631c00f570 con 0x7f632c0ff620 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.405+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c18e450 con 0x7f632c0ff620 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.405+0000 7f6322ffd700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6318038760 0x7f631803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.405+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f631c04ca50 con 0x7f632c0ff620 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.406+0000 7f63311bd700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6318038760 0x7f631803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:46.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.406+0000 7f63311bd700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6318038760 0x7f631803ac20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6328006fd0 tx=0x7f6328006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:46.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.411+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f631c02a900 con 0x7f632c0ff620 2024-03-26T22:04:46.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.678+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f632c0623e0 con 0x7f632c0ff620 2024-03-26T22:04:46.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.679+0000 7f6322ffd700 1 -- 172.21.15.196:0/2513251091 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f631c027020 con 0x7f632c0ff620 2024-03-26T22:04:46.678 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:46.678 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:46.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6318038760 msgr2=0x7f631803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:46.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6318038760 0x7f631803ac20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6328006fd0 tx=0x7f6328006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:46.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 msgr2=0x7f632c194ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:46.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c194ac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f631c006010 tx=0x7f631c009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:46.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 shutdown_connections 2024-03-26T22:04:46.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6318038760 0x7f631803ac20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:46.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 --2- 172.21.15.196:0/2513251091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f632c0ff620 0x7f632c194ac0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:46.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 >> 172.21.15.196:0/2513251091 conn(0x7f632c0fab80 msgr2=0x7f632c0fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:46.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 shutdown_connections 2024-03-26T22:04:46.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:46.681+0000 7f6333c22700 1 -- 172.21.15.196:0/2513251091 wait complete. 2024-03-26T22:04:46.682 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:47.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:46 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2513251091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:48.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:47 smithi194 ceph-mon[29693]: pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:48.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:48.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:48.503 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:48.549 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:49.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 -- 172.21.15.196:0/2336089900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 msgr2=0x7fcff40ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:49.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 --2- 172.21.15.196:0/2336089900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff40ffa50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009910 tx=0x7fcfe4009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:49.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 -- 172.21.15.196:0/2336089900 shutdown_connections 2024-03-26T22:04:49.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 --2- 172.21.15.196:0/2336089900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff40ffa50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 -- 172.21.15.196:0/2336089900 >> 172.21.15.196:0/2336089900 conn(0x7fcff40fabd0 msgr2=0x7fcff40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 -- 172.21.15.196:0/2336089900 shutdown_connections 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.982+0000 7fcffb33c700 1 -- 172.21.15.196:0/2336089900 wait complete. 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.983+0000 7fcffb33c700 1 Processor -- start 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.983+0000 7fcffb33c700 1 -- start start 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.983+0000 7fcffb33c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff4194b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.983+0000 7fcffb33c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff4195070 con 0x7fcff40ff630 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.983+0000 7fcff90d8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff4194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.983+0000 7fcff90d8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff4194b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:43228/0 (socket says 172.21.15.196:43228) 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcff90d8700 1 -- 172.21.15.196:0/2693812955 learned_addr learned my addr 172.21.15.196:0/2693812955 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:49.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcff90d8700 1 -- 172.21.15.196:0/2693812955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe40095f0 con 0x7fcff40ff630 2024-03-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcff90d8700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff4194b30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4004f40 tx=0x7fcfe4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe401d070 con 0x7fcff40ff630 2024-03-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff4195270 con 0x7fcff40ff630 2024-03-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe4005dc0 con 0x7fcff40ff630 2024-03-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe4022e40 con 0x7fcff40ff630 2024-03-26T22:04:49.986 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.984+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff4195710 con 0x7fcff40ff630 2024-03-26T22:04:49.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.985+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fcfe400b960 con 0x7fcff40ff630 2024-03-26T22:04:49.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.985+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff418e460 con 0x7fcff40ff630 2024-03-26T22:04:49.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.985+0000 7fcfea7fc700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fcfe0038760 0x7fcfe003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:49.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.985+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcfe404c870 con 0x7fcff40ff630 2024-03-26T22:04:49.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.986+0000 7fcff88d7700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fcfe0038760 0x7fcfe003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:49.987 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.986+0000 7fcff88d7700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fcfe0038760 0x7fcfe003ac20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcff0006fd0 tx=0x7fcff0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:49.990 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:49.991+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcfe400f4b0 con 0x7fcff40ff630 2024-03-26T22:04:50.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:49 smithi194 ceph-mon[29693]: pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:50.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.262+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcff40623e0 con 0x7fcff40ff630 2024-03-26T22:04:50.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.262+0000 7fcfea7fc700 1 -- 172.21.15.196:0/2693812955 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fcfe4027070 con 0x7fcff40ff630 2024-03-26T22:04:50.262 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:50.262 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:50.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fcfe0038760 msgr2=0x7fcfe003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:50.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fcfe0038760 0x7fcfe003ac20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcff0006fd0 tx=0x7fcff0006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:50.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 msgr2=0x7fcff4194b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:50.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff4194b30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4004f40 tx=0x7fcfe4004740 comp rx=0 tx=0).stop 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 shutdown_connections 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fcfe0038760 0x7fcfe003ac20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 --2- 172.21.15.196:0/2693812955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40ff630 0x7fcff4194b30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.264+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 >> 172.21.15.196:0/2693812955 conn(0x7fcff40fabd0 msgr2=0x7fcff40fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.265+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 shutdown_connections 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:50.265+0000 7fcffb33c700 1 -- 172.21.15.196:0/2693812955 wait complete. 2024-03-26T22:04:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:50.915 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:50 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2693812955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:51.790 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:51.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:51.991 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:52.037 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:52.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:51 smithi194 ceph-mon[29693]: pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 -- 172.21.15.196:0/2042504168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 msgr2=0x7f35b4100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 --2- 172.21.15.196:0/2042504168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4100b30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00b0a0 tx=0x7f35ac00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 -- 172.21.15.196:0/2042504168 shutdown_connections 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 --2- 172.21.15.196:0/2042504168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4100b30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 -- 172.21.15.196:0/2042504168 >> 172.21.15.196:0/2042504168 conn(0x7f35b40fbcd0 msgr2=0x7f35b40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 -- 172.21.15.196:0/2042504168 shutdown_connections 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.547+0000 7f35b9441700 1 -- 172.21.15.196:0/2042504168 wait complete. 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.548+0000 7f35b9441700 1 Processor -- start 2024-03-26T22:04:53.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.548+0000 7f35b9441700 1 -- start start 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.548+0000 7f35b9441700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.548+0000 7f35b9441700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b4074f60 con 0x7f35b4100710 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:50504/0 (socket says 172.21.15.196:50504) 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b2ffd700 1 -- 172.21.15.196:0/2985614427 learned_addr learned my addr 172.21.15.196:0/2985614427 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b2ffd700 1 -- 172.21.15.196:0/2985614427 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac009d00 con 0x7f35b4100710 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b2ffd700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4074a20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0062a0 tx=0x7f35ac00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac00e040 con 0x7f35b4100710 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b40731a0 con 0x7f35b4100710 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35ac003d70 con 0x7f35b4100710 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac01c890 con 0x7f35b4100710 2024-03-26T22:04:53.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.549+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b4073640 con 0x7f35b4100710 2024-03-26T22:04:53.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.550+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f35ac01cb50 con 0x7f35b4100710 2024-03-26T22:04:53.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.550+0000 7f359bfff700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f359c0387c0 0x7f359c03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:53.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.550+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b4106e70 con 0x7f35b4100710 2024-03-26T22:04:53.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.550+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f35ac04c9b0 con 0x7f35b4100710 2024-03-26T22:04:53.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.551+0000 7f35b27fc700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f359c0387c0 0x7f359c03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:53.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.551+0000 7f35b27fc700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f359c0387c0 0x7f359c03ac80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35a8006fd0 tx=0x7f35a8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:53.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.556+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f35ac003f90 con 0x7f35b4100710 2024-03-26T22:04:53.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.826+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35b40623e0 con 0x7f35b4100710 2024-03-26T22:04:53.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.826+0000 7f359bfff700 1 -- 172.21.15.196:0/2985614427 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f35ac040090 con 0x7f35b4100710 2024-03-26T22:04:53.826 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:53.826 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:53.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f359c0387c0 msgr2=0x7f359c03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:53.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f359c0387c0 0x7f359c03ac80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35a8006fd0 tx=0x7f35a8006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:53.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 msgr2=0x7f35b4074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:53.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4074a20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0062a0 tx=0x7f35ac00bd40 comp rx=0 tx=0).stop 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 shutdown_connections 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f359c0387c0 0x7f359c03ac80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 --2- 172.21.15.196:0/2985614427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35b4100710 0x7f35b4074a20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.828+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 >> 172.21.15.196:0/2985614427 conn(0x7f35b40fbcd0 msgr2=0x7f35b40fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.829+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 shutdown_connections 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:53.829+0000 7f35b9441700 1 -- 172.21.15.196:0/2985614427 wait complete. 2024-03-26T22:04:53.829 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:54.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:53 smithi194 ceph-mon[29693]: pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:55.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:54 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2985614427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:55.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:55.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:55.675 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:55.722 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:55.915 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:55 smithi194 ceph-mon[29693]: pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 -- 172.21.15.196:0/2205913621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 msgr2=0x7f223c102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 --2- 172.21.15.196:0/2205913621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c102170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f222c009910 tx=0x7f222c009c20 comp rx=0 tx=0).stop 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 -- 172.21.15.196:0/2205913621 shutdown_connections 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 --2- 172.21.15.196:0/2205913621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c102170 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 -- 172.21.15.196:0/2205913621 >> 172.21.15.196:0/2205913621 conn(0x7f223c0fa370 msgr2=0x7f223c0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 -- 172.21.15.196:0/2205913621 shutdown_connections 2024-03-26T22:04:57.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.195+0000 7f2240a7e700 1 -- 172.21.15.196:0/2205913621 wait complete. 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f2240a7e700 1 Processor -- start 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f2240a7e700 1 -- start start 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f2240a7e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f2240a7e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c196110 con 0x7f223c101d50 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f223a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f223a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:50522/0 (socket says 172.21.15.196:50522) 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f223a59c700 1 -- 172.21.15.196:0/1614140850 learned_addr learned my addr 172.21.15.196:0/1614140850 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.196+0000 7f223a59c700 1 -- 172.21.15.196:0/1614140850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0095f0 con 0x7f223c101d50 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.197+0000 7f223a59c700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c195bd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f222c006010 tx=0x7f222c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.197+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222c01d070 con 0x7f223c101d50 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.197+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c196310 con 0x7f223c101d50 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.197+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f222c005dc0 con 0x7f223c101d50 2024-03-26T22:04:57.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.197+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222c022e90 con 0x7f223c101d50 2024-03-26T22:04:57.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.197+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c1967b0 con 0x7f223c101d50 2024-03-26T22:04:57.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.198+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f222c022470 con 0x7f223c101d50 2024-03-26T22:04:57.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.198+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223c18f580 con 0x7f223c101d50 2024-03-26T22:04:57.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.198+0000 7f222b7fe700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2224038760 0x7f222403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:04:57.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.198+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f222c04c7d0 con 0x7f223c101d50 2024-03-26T22:04:57.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.198+0000 7f2239d9b700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2224038760 0x7f222403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:04:57.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.199+0000 7f2239d9b700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2224038760 0x7f222403ac20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:04:57.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.203+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f222c00b960 con 0x7f223c101d50 2024-03-26T22:04:57.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.474+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f223c0623e0 con 0x7f223c101d50 2024-03-26T22:04:57.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.475+0000 7f222b7fe700 1 -- 172.21.15.196:0/1614140850 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f223c0623e0 con 0x7f223c101d50 2024-03-26T22:04:57.475 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:04:57.475 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:04:57.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2224038760 msgr2=0x7f222403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:57.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2224038760 0x7f222403ac20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230006e40 comp rx=0 tx=0).stop 2024-03-26T22:04:57.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 msgr2=0x7f223c195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:04:57.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c195bd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f222c006010 tx=0x7f222c009f80 comp rx=0 tx=0).stop 2024-03-26T22:04:57.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 shutdown_connections 2024-03-26T22:04:57.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2224038760 0x7f222403ac20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:57.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 --2- 172.21.15.196:0/1614140850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f223c101d50 0x7f223c195bd0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:04:57.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 >> 172.21.15.196:0/1614140850 conn(0x7f223c0fa370 msgr2=0x7f223c0fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:04:57.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 shutdown_connections 2024-03-26T22:04:57.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:04:57.477+0000 7f2240a7e700 1 -- 172.21.15.196:0/1614140850 wait complete. 2024-03-26T22:04:57.477 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:04:58.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:57 smithi194 ceph-mon[29693]: pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:04:58.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:57 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1614140850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:04:59.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:04:59.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:04:59.262 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:04:59.310 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:00.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:04:59 smithi194 ceph-mon[29693]: pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:00.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 -- 172.21.15.196:0/1859508497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 msgr2=0x7f0da00ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:00.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 --2- 172.21.15.196:0/1859508497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00ffa40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009910 tx=0x7f0d9c009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 -- 172.21.15.196:0/1859508497 shutdown_connections 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 --2- 172.21.15.196:0/1859508497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00ffa40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 -- 172.21.15.196:0/1859508497 >> 172.21.15.196:0/1859508497 conn(0x7f0da00fabc0 msgr2=0x7f0da00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 -- 172.21.15.196:0/1859508497 shutdown_connections 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.759+0000 7f0da8799700 1 -- 172.21.15.196:0/1859508497 wait complete. 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da8799700 1 Processor -- start 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da8799700 1 -- start start 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da8799700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da8799700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da0069a20 con 0x7f0da00ff620 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da6535700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da6535700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:50544/0 (socket says 172.21.15.196:50544) 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da6535700 1 -- 172.21.15.196:0/1430239462 learned_addr learned my addr 172.21.15.196:0/1430239462 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.760+0000 7f0da6535700 1 -- 172.21.15.196:0/1430239462 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c0095f0 con 0x7f0da00ff620 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.761+0000 7f0da6535700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00694e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006010 tx=0x7f0d9c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.761+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c01d070 con 0x7f0da00ff620 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.761+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da0198920 con 0x7f0da00ff620 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.761+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d9c005dc0 con 0x7f0da00ff620 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.761+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da0198d60 con 0x7f0da00ff620 2024-03-26T22:05:00.762 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.761+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c022e90 con 0x7f0da00ff620 2024-03-26T22:05:00.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.762+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da018e450 con 0x7f0da00ff620 2024-03-26T22:05:00.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.762+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f0d9c022470 con 0x7f0da00ff620 2024-03-26T22:05:00.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.762+0000 7f0d977fe700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0d8c038760 0x7f0d8c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:00.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.762+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0d9c04c800 con 0x7f0da00ff620 2024-03-26T22:05:00.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.762+0000 7f0da5d34700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0d8c038760 0x7f0d8c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:00.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.763+0000 7f0da5d34700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0d8c038760 0x7f0d8c03ac20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d90006fd0 tx=0x7f0d90006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:00.767 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:00.767+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0d9c00b960 con 0x7f0da00ff620 2024-03-26T22:05:01.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.037+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0da00623e0 con 0x7f0da00ff620 2024-03-26T22:05:01.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.038+0000 7f0d977fe700 1 -- 172.21.15.196:0/1430239462 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f0d9c027030 con 0x7f0da00ff620 2024-03-26T22:05:01.037 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:01.037 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:01.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0d8c038760 msgr2=0x7f0d8c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0d8c038760 0x7f0d8c03ac20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d90006fd0 tx=0x7f0d90006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 msgr2=0x7f0da00694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00694e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006010 tx=0x7f0d9c009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 shutdown_connections 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f0d8c038760 0x7f0d8c03ac20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 --2- 172.21.15.196:0/1430239462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0da00ff620 0x7f0da00694e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 >> 172.21.15.196:0/1430239462 conn(0x7f0da00fabc0 msgr2=0x7f0da00fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 shutdown_connections 2024-03-26T22:05:01.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:01.040+0000 7f0da8799700 1 -- 172.21.15.196:0/1430239462 wait complete. 2024-03-26T22:05:01.041 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:02.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:01 smithi194 ceph-mon[29693]: pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:02.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:01 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1430239462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:02.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:02.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:02.844 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:02.889 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:04.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:03 smithi194 ceph-mon[29693]: pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:04.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.493+0000 7f3deece6700 1 -- 172.21.15.196:0/3148777567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 msgr2=0x7f3de80ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:04.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.493+0000 7f3deece6700 1 --2- 172.21.15.196:0/3148777567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de80ffa40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009910 tx=0x7f3de0009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:04.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.493+0000 7f3deece6700 1 -- 172.21.15.196:0/3148777567 shutdown_connections 2024-03-26T22:05:04.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.493+0000 7f3deece6700 1 --2- 172.21.15.196:0/3148777567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de80ffa40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:04.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.493+0000 7f3deece6700 1 -- 172.21.15.196:0/3148777567 >> 172.21.15.196:0/3148777567 conn(0x7f3de80fabc0 msgr2=0x7f3de80fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:04.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deece6700 1 -- 172.21.15.196:0/3148777567 shutdown_connections 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deece6700 1 -- 172.21.15.196:0/3148777567 wait complete. 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deece6700 1 Processor -- start 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deece6700 1 -- start start 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deece6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de8195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deece6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de81960c0 con 0x7f3de80ff620 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.494+0000 7f3deca82700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de8195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3deca82700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de8195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55726/0 (socket says 172.21.15.196:55726) 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3deca82700 1 -- 172.21.15.196:0/2864450437 learned_addr learned my addr 172.21.15.196:0/2864450437 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3deca82700 1 -- 172.21.15.196:0/2864450437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00095f0 con 0x7f3de80ff620 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3deca82700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de8195b80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3de0006010 tx=0x7f3de0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de001d070 con 0x7f3de80ff620 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de81962c0 con 0x7f3de80ff620 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3de0005dc0 con 0x7f3de80ff620 2024-03-26T22:05:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de0022e90 con 0x7f3de80ff620 2024-03-26T22:05:04.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.495+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de8196760 con 0x7f3de80ff620 2024-03-26T22:05:04.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.496+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de818f6a0 con 0x7f3de80ff620 2024-03-26T22:05:04.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.496+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3de0022470 con 0x7f3de80ff620 2024-03-26T22:05:04.497 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.496+0000 7f3de5ffb700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dd4038760 0x7f3dd403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:04.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.497+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3de004c7f0 con 0x7f3de80ff620 2024-03-26T22:05:04.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.497+0000 7f3de7fff700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dd4038760 0x7f3dd403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:04.498 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.497+0000 7f3de7fff700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dd4038760 0x7f3dd403ac20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800ad30 tx=0x7f3dd80093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:04.501 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.502+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3de000f4f0 con 0x7f3de80ff620 2024-03-26T22:05:04.769 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.769+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3de80623e0 con 0x7f3de80ff620 2024-03-26T22:05:04.769 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.770+0000 7f3de5ffb700 1 -- 172.21.15.196:0/2864450437 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3de80623e0 con 0x7f3de80ff620 2024-03-26T22:05:04.770 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:04.770 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dd4038760 msgr2=0x7f3dd403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dd4038760 0x7f3dd403ac20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800ad30 tx=0x7f3dd80093f0 comp rx=0 tx=0).stop 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 msgr2=0x7f3de8195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de8195b80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3de0006010 tx=0x7f3de0009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 shutdown_connections 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3dd4038760 0x7f3dd403ac20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 --2- 172.21.15.196:0/2864450437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3de80ff620 0x7f3de8195b80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:04.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 >> 172.21.15.196:0/2864450437 conn(0x7f3de80fabc0 msgr2=0x7f3de8068f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:04.773 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 shutdown_connections 2024-03-26T22:05:04.773 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:04.772+0000 7f3deece6700 1 -- 172.21.15.196:0/2864450437 wait complete. 2024-03-26T22:05:04.773 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:05.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:05 smithi194 ceph-mon[29693]: pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:05.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:05 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2864450437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:06.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:06.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:06.656 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:06.701 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:08.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:07 smithi194 ceph-mon[29693]: pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:08.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 -- 172.21.15.196:0/3822763750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 msgr2=0x7f9e700ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:08.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 --2- 172.21.15.196:0/3822763750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e700ffa30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009910 tx=0x7f9e60009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:08.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 -- 172.21.15.196:0/3822763750 shutdown_connections 2024-03-26T22:05:08.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 --2- 172.21.15.196:0/3822763750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e700ffa30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:08.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 -- 172.21.15.196:0/3822763750 >> 172.21.15.196:0/3822763750 conn(0x7f9e700fabb0 msgr2=0x7f9e700fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:08.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 -- 172.21.15.196:0/3822763750 shutdown_connections 2024-03-26T22:05:08.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 -- 172.21.15.196:0/3822763750 wait complete. 2024-03-26T22:05:08.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.173+0000 7f9e75c6a700 1 Processor -- start 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e75c6a700 1 -- start start 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e75c6a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e70069540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e75c6a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70069a80 con 0x7f9e700ff610 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e6f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e70069540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e6f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e70069540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55732/0 (socket says 172.21.15.196:55732) 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e6f7fe700 1 -- 172.21.15.196:0/1465839928 learned_addr learned my addr 172.21.15.196:0/1465839928 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e6f7fe700 1 -- 172.21.15.196:0/1465839928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600095f0 con 0x7f9e700ff610 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.174+0000 7f9e6f7fe700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e70069540 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9e60006010 tx=0x7f9e60009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.175+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e6001d070 con 0x7f9e700ff610 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.175+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e70198930 con 0x7f9e700ff610 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.175+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e60005dc0 con 0x7f9e700ff610 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.175+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e60022e90 con 0x7f9e700ff610 2024-03-26T22:05:08.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.175+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e70198d70 con 0x7f9e700ff610 2024-03-26T22:05:08.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.176+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9e60022470 con 0x7f9e700ff610 2024-03-26T22:05:08.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.176+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7018e440 con 0x7f9e700ff610 2024-03-26T22:05:08.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.176+0000 7f9e6cff9700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9e58038760 0x7f9e5803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:08.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.176+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9e6004c8b0 con 0x7f9e700ff610 2024-03-26T22:05:08.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.176+0000 7f9e6effd700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9e58038760 0x7f9e5803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:08.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.176+0000 7f9e6effd700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9e58038760 0x7f9e5803ac20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e64006fd0 tx=0x7f9e64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:08.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.181+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e6000b960 con 0x7f9e700ff610 2024-03-26T22:05:08.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.454+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e700623e0 con 0x7f9e700ff610 2024-03-26T22:05:08.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.455+0000 7f9e6cff9700 1 -- 172.21.15.196:0/1465839928 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9e60027030 con 0x7f9e700ff610 2024-03-26T22:05:08.456 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:08.456 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9e58038760 msgr2=0x7f9e5803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9e58038760 0x7f9e5803ac20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e64006fd0 tx=0x7f9e64006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 msgr2=0x7f9e70069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e70069540 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9e60006010 tx=0x7f9e60009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 shutdown_connections 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9e58038760 0x7f9e5803ac20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:08.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 --2- 172.21.15.196:0/1465839928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e700ff610 0x7f9e70069540 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:08.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 >> 172.21.15.196:0/1465839928 conn(0x7f9e700fabb0 msgr2=0x7f9e700fb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:08.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 shutdown_connections 2024-03-26T22:05:08.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:08.457+0000 7f9e75c6a700 1 -- 172.21.15.196:0/1465839928 wait complete. 2024-03-26T22:05:08.458 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:09.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:08 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1465839928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:10.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:10.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:10.066 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:09 smithi194 ceph-mon[29693]: pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:10.225 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:10.271 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:11.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.832+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/1145966834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 msgr2=0x7fe9a0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:11.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.832+0000 7fe9a4cd1700 1 --2- 172.21.15.196:0/1145966834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0100b40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe994009910 tx=0x7fe994009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:11.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/1145966834 shutdown_connections 2024-03-26T22:05:11.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 --2- 172.21.15.196:0/1145966834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0100b40 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:11.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/1145966834 >> 172.21.15.196:0/1145966834 conn(0x7fe9a00fbcc0 msgr2=0x7fe9a00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:11.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/1145966834 shutdown_connections 2024-03-26T22:05:11.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/1145966834 wait complete. 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 Processor -- start 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 -- start start 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.833+0000 7fe9a4cd1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a0196100 con 0x7fe9a0100720 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe99e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe99e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:55748/0 (socket says 172.21.15.196:55748) 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe99e59c700 1 -- 172.21.15.196:0/2782623895 learned_addr learned my addr 172.21.15.196:0/2782623895 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe99e59c700 1 -- 172.21.15.196:0/2782623895 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940095f0 con 0x7fe9a0100720 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe99e59c700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0195bc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe994006010 tx=0x7fe994009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe99401d070 con 0x7fe9a0100720 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a0196300 con 0x7fe9a0100720 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe994005dc0 con 0x7fe9a0100720 2024-03-26T22:05:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.834+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe994022e90 con 0x7fe9a0100720 2024-03-26T22:05:11.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.835+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a01967a0 con 0x7fe9a0100720 2024-03-26T22:05:11.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.835+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fe994022470 con 0x7fe9a0100720 2024-03-26T22:05:11.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.835+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a018f550 con 0x7fe9a0100720 2024-03-26T22:05:11.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.836+0000 7fe98f7fe700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe988038760 0x7fe98803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:11.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.836+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe99404c7d0 con 0x7fe9a0100720 2024-03-26T22:05:11.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.836+0000 7fe99dd9b700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe988038760 0x7fe98803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:11.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.836+0000 7fe99dd9b700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe988038760 0x7fe98803ac20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe990006fd0 tx=0x7fe990006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:11.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:11.841+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe99400b960 con 0x7fe9a0100720 2024-03-26T22:05:12.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.110+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9a00623e0 con 0x7fe9a0100720 2024-03-26T22:05:12.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.111+0000 7fe98f7fe700 1 -- 172.21.15.196:0/2782623895 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe9a00623e0 con 0x7fe9a0100720 2024-03-26T22:05:12.111 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:12.111 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe988038760 msgr2=0x7fe98803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe988038760 0x7fe98803ac20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe990006fd0 tx=0x7fe990006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 msgr2=0x7fe9a0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0195bc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe994006010 tx=0x7fe994009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 shutdown_connections 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fe988038760 0x7fe98803ac20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 --2- 172.21.15.196:0/2782623895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9a0100720 0x7fe9a0195bc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 >> 172.21.15.196:0/2782623895 conn(0x7fe9a00fbcc0 msgr2=0x7fe9a00fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:12.114 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 shutdown_connections 2024-03-26T22:05:12.114 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:12.113+0000 7fe9a4cd1700 1 -- 172.21.15.196:0/2782623895 wait complete. 2024-03-26T22:05:12.114 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:12.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:11 smithi194 ceph-mon[29693]: pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:13.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:12 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2782623895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:13.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:13.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:13.899 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:13.945 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:14.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:13 smithi194 ceph-mon[29693]: pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:15.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:15 smithi194 ceph-mon[29693]: pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:15.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 -- 172.21.15.196:0/1065115870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 msgr2=0x7f49c80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:15.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 --2- 172.21.15.196:0/1065115870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c80731e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009910 tx=0x7f49b8009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:15.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 -- 172.21.15.196:0/1065115870 shutdown_connections 2024-03-26T22:05:15.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 --2- 172.21.15.196:0/1065115870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c80731e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:15.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 -- 172.21.15.196:0/1065115870 >> 172.21.15.196:0/1065115870 conn(0x7f49c80fbcc0 msgr2=0x7f49c80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:15.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 -- 172.21.15.196:0/1065115870 shutdown_connections 2024-03-26T22:05:15.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 -- 172.21.15.196:0/1065115870 wait complete. 2024-03-26T22:05:15.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 Processor -- start 2024-03-26T22:05:15.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.484+0000 7f49cdba1700 1 -- start start 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49cdba1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c8197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49cdba1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c81982e0 con 0x7f49c8074d80 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49c77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c8197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49c77fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c8197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39556/0 (socket says 172.21.15.196:39556) 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49c77fe700 1 -- 172.21.15.196:0/1845927907 learned_addr learned my addr 172.21.15.196:0/1845927907 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49c77fe700 1 -- 172.21.15.196:0/1845927907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80095f0 con 0x7f49c8074d80 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.485+0000 7f49c77fe700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c8197da0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009ed0 tx=0x7f49b8004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49b801d070 con 0x7f49c8074d80 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49b800be20 con 0x7f49c8074d80 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c81984e0 con 0x7f49c8074d80 2024-03-26T22:05:15.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49b800f460 con 0x7f49c8074d80 2024-03-26T22:05:15.487 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c8198980 con 0x7f49c8074d80 2024-03-26T22:05:15.490 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c8191750 con 0x7f49c8074d80 2024-03-26T22:05:15.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.486+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f49b8022800 con 0x7f49c8074d80 2024-03-26T22:05:15.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.487+0000 7f49c4ff9700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49b0038820 0x7f49b003ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:15.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.487+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f49b804cbb0 con 0x7f49c8074d80 2024-03-26T22:05:15.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.487+0000 7f49c6ffd700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49b0038820 0x7f49b003ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:15.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.487+0000 7f49c6ffd700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49b0038820 0x7f49b003ace0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49bc006fd0 tx=0x7f49bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:15.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.492+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49b802ba00 con 0x7f49c8074d80 2024-03-26T22:05:15.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.760+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49c80623e0 con 0x7f49c8074d80 2024-03-26T22:05:15.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.761+0000 7f49c4ff9700 1 -- 172.21.15.196:0/1845927907 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f49c80623e0 con 0x7f49c8074d80 2024-03-26T22:05:15.761 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:15.761 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49b0038820 msgr2=0x7f49b003ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49b0038820 0x7f49b003ace0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49bc006fd0 tx=0x7f49bc006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 msgr2=0x7f49c8197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c8197da0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009ed0 tx=0x7f49b8004690 comp rx=0 tx=0).stop 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 shutdown_connections 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f49b0038820 0x7f49b003ace0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 --2- 172.21.15.196:0/1845927907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49c8074d80 0x7f49c8197da0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 >> 172.21.15.196:0/1845927907 conn(0x7f49c80fbcc0 msgr2=0x7f49c80fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 shutdown_connections 2024-03-26T22:05:15.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:15.763+0000 7f49cdba1700 1 -- 172.21.15.196:0/1845927907 wait complete. 2024-03-26T22:05:15.764 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:16.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:16 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1845927907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:17.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:17.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:17.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:17 smithi194 ceph-mon[29693]: pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:17.573 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:17.619 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:19.131 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 -- 172.21.15.196:0/3141246774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 msgr2=0x7f88380ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 --2- 172.21.15.196:0/3141246774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380ffa40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8828009910 tx=0x7f8828009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 -- 172.21.15.196:0/3141246774 shutdown_connections 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 --2- 172.21.15.196:0/3141246774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380ffa40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 -- 172.21.15.196:0/3141246774 >> 172.21.15.196:0/3141246774 conn(0x7f88380fab80 msgr2=0x7f88380fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 -- 172.21.15.196:0/3141246774 shutdown_connections 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.130+0000 7f8840682700 1 -- 172.21.15.196:0/3141246774 wait complete. 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f8840682700 1 Processor -- start 2024-03-26T22:05:19.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f8840682700 1 -- start start 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f8840682700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f8840682700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8838068e10 con 0x7f88380ff620 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f883e41e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f883e41e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380688d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39572/0 (socket says 172.21.15.196:39572) 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.131+0000 7f883e41e700 1 -- 172.21.15.196:0/3050848046 learned_addr learned my addr 172.21.15.196:0/3050848046 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f883e41e700 1 -- 172.21.15.196:0/3050848046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280095f0 con 0x7f88380ff620 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f883e41e700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380688d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8828006010 tx=0x7f8828009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882801d070 con 0x7f88380ff620 2024-03-26T22:05:19.133 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8838198910 con 0x7f88380ff620 2024-03-26T22:05:19.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8828005dc0 con 0x7f88380ff620 2024-03-26T22:05:19.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8828022e90 con 0x7f88380ff620 2024-03-26T22:05:19.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.132+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8838198d50 con 0x7f88380ff620 2024-03-26T22:05:19.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.133+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883818e450 con 0x7f88380ff620 2024-03-26T22:05:19.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.133+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f8828022470 con 0x7f88380ff620 2024-03-26T22:05:19.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.133+0000 7f882f7fe700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8824038760 0x7f882403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:19.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.133+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f882804c7f0 con 0x7f88380ff620 2024-03-26T22:05:19.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.133+0000 7f883dc1d700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8824038760 0x7f882403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:19.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.134+0000 7f883dc1d700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8824038760 0x7f882403ac20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8834006fd0 tx=0x7f8834006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:19.138 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.138+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f882800b960 con 0x7f88380ff620 2024-03-26T22:05:19.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.405+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88380623e0 con 0x7f88380ff620 2024-03-26T22:05:19.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.406+0000 7f882f7fe700 1 -- 172.21.15.196:0/3050848046 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8828027030 con 0x7f88380ff620 2024-03-26T22:05:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:19.406 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8824038760 msgr2=0x7f882403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8824038760 0x7f882403ac20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8834006fd0 tx=0x7f8834006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 msgr2=0x7f88380688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380688d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8828006010 tx=0x7f8828009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 shutdown_connections 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f8824038760 0x7f882403ac20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 --2- 172.21.15.196:0/3050848046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88380ff620 0x7f88380688d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:19.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 >> 172.21.15.196:0/3050848046 conn(0x7f88380fab80 msgr2=0x7f88380fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:19.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 shutdown_connections 2024-03-26T22:05:19.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:19.408+0000 7f8840682700 1 -- 172.21.15.196:0/3050848046 wait complete. 2024-03-26T22:05:19.409 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:20.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:19 smithi194 ceph-mon[29693]: pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:20.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:19 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3050848046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:21.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:21.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:21.301 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:21.347 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:22.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:21 smithi194 ceph-mon[29693]: pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.895+0000 7fda509e6700 1 -- 172.21.15.196:0/1671940402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 msgr2=0x7fda4c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.895+0000 7fda509e6700 1 --2- 172.21.15.196:0/1671940402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c0ffa30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fda4400b0a0 tx=0x7fda4400b3b0 comp rx=0 tx=0).stop 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.895+0000 7fda509e6700 1 -- 172.21.15.196:0/1671940402 shutdown_connections 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.895+0000 7fda509e6700 1 --2- 172.21.15.196:0/1671940402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c0ffa30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.895+0000 7fda509e6700 1 -- 172.21.15.196:0/1671940402 >> 172.21.15.196:0/1671940402 conn(0x7fda4c0fabb0 msgr2=0x7fda4c0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.896+0000 7fda509e6700 1 -- 172.21.15.196:0/1671940402 shutdown_connections 2024-03-26T22:05:22.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.896+0000 7fda509e6700 1 -- 172.21.15.196:0/1671940402 wait complete. 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.896+0000 7fda509e6700 1 Processor -- start 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.896+0000 7fda509e6700 1 -- start start 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda509e6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c194ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda509e6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c194ff0 con 0x7fda4c0ff610 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda4a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c194ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda4a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c194ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39524/0 (socket says 172.21.15.196:39524) 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda4a59c700 1 -- 172.21.15.196:0/3506610441 learned_addr learned my addr 172.21.15.196:0/3506610441 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda4a59c700 1 -- 172.21.15.196:0/3506610441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda44009d00 con 0x7fda4c0ff610 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.897+0000 7fda4a59c700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c194ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fda440062a0 tx=0x7fda4400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.898+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4400e040 con 0x7fda4c0ff610 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.898+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c1951f0 con 0x7fda4c0ff610 2024-03-26T22:05:22.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.898+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda44003d70 con 0x7fda4c0ff610 2024-03-26T22:05:22.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.898+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4401c890 con 0x7fda4c0ff610 2024-03-26T22:05:22.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.898+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c195690 con 0x7fda4c0ff610 2024-03-26T22:05:22.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.899+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c18e440 con 0x7fda4c0ff610 2024-03-26T22:05:22.901 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.899+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fda4401cb50 con 0x7fda4c0ff610 2024-03-26T22:05:22.901 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.899+0000 7fda437fe700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fda380387c0 0x7fda3803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:22.901 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.899+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fda4404c8e0 con 0x7fda4c0ff610 2024-03-26T22:05:22.901 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.899+0000 7fda49d9b700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fda380387c0 0x7fda3803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:22.901 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.899+0000 7fda49d9b700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fda380387c0 0x7fda3803ac80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda34006fd0 tx=0x7fda34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:22.904 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:22.904+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fda4401a590 con 0x7fda4c0ff610 2024-03-26T22:05:23.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.177+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda4c0623e0 con 0x7fda4c0ff610 2024-03-26T22:05:23.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.177+0000 7fda437fe700 1 -- 172.21.15.196:0/3506610441 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fda4c0623e0 con 0x7fda4c0ff610 2024-03-26T22:05:23.177 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:23.177 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:23.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fda380387c0 msgr2=0x7fda3803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:23.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fda380387c0 0x7fda3803ac80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda34006fd0 tx=0x7fda34006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:23.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 msgr2=0x7fda4c194ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:23.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c194ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fda440062a0 tx=0x7fda4400bd40 comp rx=0 tx=0).stop 2024-03-26T22:05:23.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 shutdown_connections 2024-03-26T22:05:23.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fda380387c0 0x7fda3803ac80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:23.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 --2- 172.21.15.196:0/3506610441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda4c0ff610 0x7fda4c194ab0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:23.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 >> 172.21.15.196:0/3506610441 conn(0x7fda4c0fabb0 msgr2=0x7fda4c0fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:23.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 shutdown_connections 2024-03-26T22:05:23.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:23.179+0000 7fda509e6700 1 -- 172.21.15.196:0/3506610441 wait complete. 2024-03-26T22:05:23.180 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:24.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:23 smithi194 ceph-mon[29693]: pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:24.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:23 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3506610441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:24.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:24.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:25.020 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:25.066 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:25.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:25 smithi194 ceph-mon[29693]: pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:26.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 -- 172.21.15.196:0/4034251026 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 msgr2=0x7f22b4100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:26.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 --2- 172.21.15.196:0/4034251026 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4100b30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00b0a0 tx=0x7f22ac00b3b0 comp rx=0 tx=0).stop 2024-03-26T22:05:26.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 -- 172.21.15.196:0/4034251026 shutdown_connections 2024-03-26T22:05:26.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 --2- 172.21.15.196:0/4034251026 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4100b30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:26.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 -- 172.21.15.196:0/4034251026 >> 172.21.15.196:0/4034251026 conn(0x7f22b40fbcd0 msgr2=0x7f22b40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:26.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 -- 172.21.15.196:0/4034251026 shutdown_connections 2024-03-26T22:05:26.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.646+0000 7f22ba21d700 1 -- 172.21.15.196:0/4034251026 wait complete. 2024-03-26T22:05:26.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22ba21d700 1 Processor -- start 2024-03-26T22:05:26.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22ba21d700 1 -- start start 2024-03-26T22:05:26.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22ba21d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22ba21d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b41960f0 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22b37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22b37fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4195bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39554/0 (socket says 172.21.15.196:39554) 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22b37fe700 1 -- 172.21.15.196:0/4256942967 learned_addr learned my addr 172.21.15.196:0/4256942967 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.647+0000 7f22b37fe700 1 -- 172.21.15.196:0/4256942967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac009d00 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.648+0000 7f22b37fe700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4195bb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0062a0 tx=0x7f22ac00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.648+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac00e040 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.648+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22ac003d70 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.648+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b41962f0 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.648+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac01c890 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.648+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b4196790 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.649+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b418f540 con 0x7f22b4100710 2024-03-26T22:05:26.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.649+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f22ac01cb50 con 0x7f22b4100710 2024-03-26T22:05:26.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.649+0000 7f22b0ff9700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f22a00387c0 0x7f22a003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:26.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.649+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f22ac04c9e0 con 0x7f22b4100710 2024-03-26T22:05:26.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.649+0000 7f22b2ffd700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f22a00387c0 0x7f22a003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:26.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.649+0000 7f22b2ffd700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f22a00387c0 0x7f22a003ac80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f229c006fd0 tx=0x7f229c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:26.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.654+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f22ac01a3b0 con 0x7f22b4100710 2024-03-26T22:05:26.930 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.931+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22b40623e0 con 0x7f22b4100710 2024-03-26T22:05:26.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.931+0000 7f22b0ff9700 1 -- 172.21.15.196:0/4256942967 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f22b40623e0 con 0x7f22b4100710 2024-03-26T22:05:26.931 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:26.932 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:26.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f22a00387c0 msgr2=0x7f22a003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:26.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f22a00387c0 0x7f22a003ac80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f229c006fd0 tx=0x7f229c006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:26.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 msgr2=0x7f22b4195bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:26.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4195bb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0062a0 tx=0x7f22ac00bd40 comp rx=0 tx=0).stop 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 shutdown_connections 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f22a00387c0 0x7f22a003ac80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 --2- 172.21.15.196:0/4256942967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22b4100710 0x7f22b4195bb0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 >> 172.21.15.196:0/4256942967 conn(0x7f22b40fbcd0 msgr2=0x7f22b40fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 shutdown_connections 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:26.934+0000 7f22ba21d700 1 -- 172.21.15.196:0/4256942967 wait complete. 2024-03-26T22:05:26.934 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:28.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:27 smithi194 ceph-mon[29693]: pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:28.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:27 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/4256942967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:28.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:28.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:28.738 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:28.783 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:30.166 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:29 smithi194 ceph-mon[29693]: pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.301+0000 7f2151ad8700 1 -- 172.21.15.196:0/3279111625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 msgr2=0x7f214c100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.301+0000 7f2151ad8700 1 --2- 172.21.15.196:0/3279111625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c100b30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f213c009910 tx=0x7f213c009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.301+0000 7f2151ad8700 1 -- 172.21.15.196:0/3279111625 shutdown_connections 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.301+0000 7f2151ad8700 1 --2- 172.21.15.196:0/3279111625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c100b30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.301+0000 7f2151ad8700 1 -- 172.21.15.196:0/3279111625 >> 172.21.15.196:0/3279111625 conn(0x7f214c0fbcd0 msgr2=0x7f214c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.302+0000 7f2151ad8700 1 -- 172.21.15.196:0/3279111625 shutdown_connections 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.302+0000 7f2151ad8700 1 -- 172.21.15.196:0/3279111625 wait complete. 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.302+0000 7f2151ad8700 1 Processor -- start 2024-03-26T22:05:30.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.302+0000 7f2151ad8700 1 -- start start 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.302+0000 7f2151ad8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.302+0000 7f2151ad8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c074f60 con 0x7f214c100710 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.303+0000 7f214b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.303+0000 7f214b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:39576/0 (socket says 172.21.15.196:39576) 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.303+0000 7f214b7fe700 1 -- 172.21.15.196:0/1309847847 learned_addr learned my addr 172.21.15.196:0/1309847847 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.303+0000 7f214b7fe700 1 -- 172.21.15.196:0/1309847847 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c0095f0 con 0x7f214c100710 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.303+0000 7f214b7fe700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c074a20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f213c006010 tx=0x7f213c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.303+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f213c01d070 con 0x7f214c100710 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.304+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f213c005dc0 con 0x7f214c100710 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.304+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214c0731a0 con 0x7f214c100710 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.304+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f213c022e90 con 0x7f214c100710 2024-03-26T22:05:30.304 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.304+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214c073640 con 0x7f214c100710 2024-03-26T22:05:30.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.304+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f213c00f4f0 con 0x7f214c100710 2024-03-26T22:05:30.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.305+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214c106e70 con 0x7f214c100710 2024-03-26T22:05:30.305 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.305+0000 7f21497fa700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2138038760 0x7f213803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:30.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.305+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f213c04caa0 con 0x7f214c100710 2024-03-26T22:05:30.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.305+0000 7f2142ffd700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2138038760 0x7f213803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:30.306 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.305+0000 7f2142ffd700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2138038760 0x7f213803ac20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2134006fd0 tx=0x7f2134006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:30.310 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.310+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f213c02a950 con 0x7f214c100710 2024-03-26T22:05:30.576 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.577+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f214c0623e0 con 0x7f214c100710 2024-03-26T22:05:30.577 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.577+0000 7f21497fa700 1 -- 172.21.15.196:0/1309847847 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f213c027020 con 0x7f214c100710 2024-03-26T22:05:30.577 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:30.577 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:30.579 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2138038760 msgr2=0x7f213803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:30.579 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2138038760 0x7f213803ac20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2134006fd0 tx=0x7f2134006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:30.579 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 msgr2=0x7f214c074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:30.579 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c074a20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f213c006010 tx=0x7f213c009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:30.579 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 shutdown_connections 2024-03-26T22:05:30.579 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f2138038760 0x7f213803ac20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:30.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 --2- 172.21.15.196:0/1309847847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f214c100710 0x7f214c074a20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:30.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.579+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 >> 172.21.15.196:0/1309847847 conn(0x7f214c0fbcd0 msgr2=0x7f214c0fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:30.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.580+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 shutdown_connections 2024-03-26T22:05:30.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:30.580+0000 7f2151ad8700 1 -- 172.21.15.196:0/1309847847 wait complete. 2024-03-26T22:05:30.580 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:31.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:30 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1309847847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:32.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:32.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:32.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:31 smithi194 ceph-mon[29693]: pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:32.304 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:32.350 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:33.874 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 -- 172.21.15.196:0/1473904732 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 msgr2=0x7fbf280ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 --2- 172.21.15.196:0/1473904732 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280ffa40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009910 tx=0x7fbf1c009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 -- 172.21.15.196:0/1473904732 shutdown_connections 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 --2- 172.21.15.196:0/1473904732 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280ffa40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 -- 172.21.15.196:0/1473904732 >> 172.21.15.196:0/1473904732 conn(0x7fbf280fabc0 msgr2=0x7fbf280fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 -- 172.21.15.196:0/1473904732 shutdown_connections 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.873+0000 7fbf2edfe700 1 -- 172.21.15.196:0/1473904732 wait complete. 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2edfe700 1 Processor -- start 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2edfe700 1 -- start start 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2edfe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2edfe700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280fcf70 con 0x7fbf280ff620 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2cb9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2cb9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45760/0 (socket says 172.21.15.196:45760) 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.874+0000 7fbf2cb9a700 1 -- 172.21.15.196:0/2854369969 learned_addr learned my addr 172.21.15.196:0/2854369969 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf2cb9a700 1 -- 172.21.15.196:0/2854369969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c0095f0 con 0x7fbf280ff620 2024-03-26T22:05:33.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf2cb9a700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280fca30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c006010 tx=0x7fbf1c0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:33.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1c01d070 con 0x7fbf280ff620 2024-03-26T22:05:33.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf1c00be20 con 0x7fbf280ff620 2024-03-26T22:05:33.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf280fb1b0 con 0x7fbf280ff620 2024-03-26T22:05:33.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1c022de0 con 0x7fbf280ff620 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.875+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf280fb650 con 0x7fbf280ff620 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.876+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf28105d80 con 0x7fbf280ff620 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.876+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fbf1c00b960 con 0x7fbf280ff620 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.876+0000 7fbf25ffb700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbf180387b0 0x7fbf1803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.876+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbf1c04c8c0 con 0x7fbf280ff620 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.876+0000 7fbf27fff700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbf180387b0 0x7fbf1803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:33.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.877+0000 7fbf27fff700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbf180387b0 0x7fbf1803ac70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf14006fd0 tx=0x7fbf14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:33.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:33.882+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbf1c0226c0 con 0x7fbf280ff620 2024-03-26T22:05:34.149 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.149+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf280623e0 con 0x7fbf280ff620 2024-03-26T22:05:34.149 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.150+0000 7fbf25ffb700 1 -- 172.21.15.196:0/2854369969 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbf1c027030 con 0x7fbf280ff620 2024-03-26T22:05:34.150 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:34.150 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:34.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbf180387b0 msgr2=0x7fbf1803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:34.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbf180387b0 0x7fbf1803ac70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf14006fd0 tx=0x7fbf14006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:34.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 msgr2=0x7fbf280fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:34.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280fca30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c006010 tx=0x7fbf1c0050b0 comp rx=0 tx=0).stop 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 shutdown_connections 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbf180387b0 0x7fbf1803ac70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 --2- 172.21.15.196:0/2854369969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf280ff620 0x7fbf280fca30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 >> 172.21.15.196:0/2854369969 conn(0x7fbf280fabc0 msgr2=0x7fbf28068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 shutdown_connections 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:34.152+0000 7fbf2edfe700 1 -- 172.21.15.196:0/2854369969 wait complete. 2024-03-26T22:05:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:34.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:33 smithi194 ceph-mon[29693]: pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:35.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:35 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2854369969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:35.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:35 smithi194 ceph-mon[29693]: pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:35.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:35.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:35.953 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:35.999 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.542+0000 7fa21c2b7700 1 -- 172.21.15.196:0/2955320582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 msgr2=0x7fa2140ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.542+0000 7fa21c2b7700 1 --2- 172.21.15.196:0/2955320582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140ffa80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa204009910 tx=0x7fa204009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 -- 172.21.15.196:0/2955320582 shutdown_connections 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 --2- 172.21.15.196:0/2955320582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140ffa80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 -- 172.21.15.196:0/2955320582 >> 172.21.15.196:0/2955320582 conn(0x7fa2140fac20 msgr2=0x7fa2140fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 -- 172.21.15.196:0/2955320582 shutdown_connections 2024-03-26T22:05:37.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 -- 172.21.15.196:0/2955320582 wait complete. 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 Processor -- start 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 -- start start 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140fca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21c2b7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140fcf80 con 0x7fa2140ff660 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.543+0000 7fa21a053700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa21a053700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140fca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45764/0 (socket says 172.21.15.196:45764) 2024-03-26T22:05:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa21a053700 1 -- 172.21.15.196:0/3236001314 learned_addr learned my addr 172.21.15.196:0/3236001314 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:37.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa21a053700 1 -- 172.21.15.196:0/3236001314 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2040095f0 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa21a053700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140fca40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa204004f40 tx=0x7fa204004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20401d070 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2140fb1c0 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa204005dc0 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204022e40 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.544+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2140fb660 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.545+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa214105dc0 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.545+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa2040226c0 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.545+0000 7fa20b7fe700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa200038760 0x7fa20003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.545+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa20404c980 con 0x7fa2140ff660 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.545+0000 7fa219852700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa200038760 0x7fa20003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.546+0000 7fa219852700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa200038760 0x7fa20003ac20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa210006fd0 tx=0x7fa210006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:37.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.550+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa20400b960 con 0x7fa2140ff660 2024-03-26T22:05:37.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.820+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2140623e0 con 0x7fa2140ff660 2024-03-26T22:05:37.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.820+0000 7fa20b7fe700 1 -- 172.21.15.196:0/3236001314 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa204027030 con 0x7fa2140ff660 2024-03-26T22:05:37.820 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:37.820 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:37.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa200038760 msgr2=0x7fa20003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:37.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa200038760 0x7fa20003ac20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa210006fd0 tx=0x7fa210006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 msgr2=0x7fa2140fca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140fca40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa204004f40 tx=0x7fa204004740 comp rx=0 tx=0).stop 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 shutdown_connections 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa200038760 0x7fa20003ac20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 --2- 172.21.15.196:0/3236001314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2140ff660 0x7fa2140fca40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.822+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 >> 172.21.15.196:0/3236001314 conn(0x7fa2140fac20 msgr2=0x7fa214068f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.823+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 shutdown_connections 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:37.823+0000 7fa21c2b7700 1 -- 172.21.15.196:0/3236001314 wait complete. 2024-03-26T22:05:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:38.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:37 smithi194 ceph-mon[29693]: pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:39.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:38 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3236001314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:39.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:39.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:39.664 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:39.710 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:40.066 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:39 smithi194 ceph-mon[29693]: pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1145425348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 msgr2=0x7f9d840ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 --2- 172.21.15.196:0/1145425348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d840ffa30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009910 tx=0x7f9d80009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1145425348 shutdown_connections 2024-03-26T22:05:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 --2- 172.21.15.196:0/1145425348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d840ffa30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1145425348 >> 172.21.15.196:0/1145425348 conn(0x7f9d840fabb0 msgr2=0x7f9d840fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1145425348 shutdown_connections 2024-03-26T22:05:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.196+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1145425348 wait complete. 2024-03-26T22:05:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d8be2b700 1 Processor -- start 2024-03-26T22:05:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d8be2b700 1 -- start start 2024-03-26T22:05:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d8be2b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d84195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d8be2b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84196110 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d89bc7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d84195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d89bc7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d84195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:45786/0 (socket says 172.21.15.196:45786) 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d89bc7700 1 -- 172.21.15.196:0/1320031174 learned_addr learned my addr 172.21.15.196:0/1320031174 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.197+0000 7f9d89bc7700 1 -- 172.21.15.196:0/1320031174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d800095f0 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.198+0000 7f9d89bc7700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d84195bd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d80006010 tx=0x7f9d80009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.198+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d8001d070 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.198+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d80005dc0 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.198+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d80022e90 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.198+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d84196310 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.198+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d841967b0 con 0x7f9d840ff610 2024-03-26T22:05:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.199+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d840623e0 con 0x7f9d840ff610 2024-03-26T22:05:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.199+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f9d80022470 con 0x7f9d840ff610 2024-03-26T22:05:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.199+0000 7f9d7affd700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9d70038760 0x7f9d7003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.199+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d8004c7f0 con 0x7f9d840ff610 2024-03-26T22:05:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.199+0000 7f9d893c6700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9d70038760 0x7f9d7003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.200+0000 7f9d893c6700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9d70038760 0x7f9d7003ac20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9d74006fd0 tx=0x7f9d74006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.204+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9d8000b960 con 0x7f9d840ff610 2024-03-26T22:05:41.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.470+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d84105640 con 0x7f9d840ff610 2024-03-26T22:05:41.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.470+0000 7f9d7affd700 1 -- 172.21.15.196:0/1320031174 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9d80027030 con 0x7f9d840ff610 2024-03-26T22:05:41.470 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:41.470 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:41.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9d70038760 msgr2=0x7f9d7003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:41.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9d70038760 0x7f9d7003ac20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9d74006fd0 tx=0x7f9d74006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:41.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 msgr2=0x7f9d84195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d84195bd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d80006010 tx=0x7f9d80009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 shutdown_connections 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f9d70038760 0x7f9d7003ac20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 --2- 172.21.15.196:0/1320031174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d840ff610 0x7f9d84195bd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 >> 172.21.15.196:0/1320031174 conn(0x7f9d840fabb0 msgr2=0x7f9d84068730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 shutdown_connections 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:41.473+0000 7f9d8be2b700 1 -- 172.21.15.196:0/1320031174 wait complete. 2024-03-26T22:05:41.474 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:42.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:41 smithi194 ceph-mon[29693]: pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:42.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:41 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1320031174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:43.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:43.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:43.226 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:43.272 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:44.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:43 smithi194 ceph-mon[29693]: pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:44.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 -- 172.21.15.196:0/651151301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 msgr2=0x7fed50100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:44.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 --2- 172.21.15.196:0/651151301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50100b40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fed4800b0a0 tx=0x7fed4800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:05:44.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 -- 172.21.15.196:0/651151301 shutdown_connections 2024-03-26T22:05:44.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 --2- 172.21.15.196:0/651151301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50100b40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:44.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 -- 172.21.15.196:0/651151301 >> 172.21.15.196:0/651151301 conn(0x7fed500fbcc0 msgr2=0x7fed500fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:44.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 -- 172.21.15.196:0/651151301 shutdown_connections 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.882+0000 7fed573b9700 1 -- 172.21.15.196:0/651151301 wait complete. 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed573b9700 1 Processor -- start 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed573b9700 1 -- start start 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed573b9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed573b9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed50196100 con 0x7fed50100720 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed55155700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed55155700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60062/0 (socket says 172.21.15.196:60062) 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed55155700 1 -- 172.21.15.196:0/1791462514 learned_addr learned my addr 172.21.15.196:0/1791462514 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.883+0000 7fed55155700 1 -- 172.21.15.196:0/1791462514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed48009d00 con 0x7fed50100720 2024-03-26T22:05:44.884 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.884+0000 7fed55155700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50195bc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fed480062a0 tx=0x7fed4800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:44.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.884+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed4800e040 con 0x7fed50100720 2024-03-26T22:05:44.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.884+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed50196300 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.884+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed48003d70 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.884+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed4801c890 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.884+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed501967a0 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.885+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fed4801cb50 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.885+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5018f550 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.885+0000 7fed4e7fc700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fed380387c0 0x7fed3803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.885+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fed4804c9e0 con 0x7fed50100720 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.885+0000 7fed54954700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fed380387c0 0x7fed3803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.885+0000 7fed54954700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fed380387c0 0x7fed3803ac80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed44006fd0 tx=0x7fed44006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:44.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:44.890+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fed4801a3b0 con 0x7fed50100720 2024-03-26T22:05:45.161 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.161+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed500623e0 con 0x7fed50100720 2024-03-26T22:05:45.161 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.162+0000 7fed4e7fc700 1 -- 172.21.15.196:0/1791462514 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fed500623e0 con 0x7fed50100720 2024-03-26T22:05:45.161 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:45.161 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:45.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fed380387c0 msgr2=0x7fed3803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fed380387c0 0x7fed3803ac80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed44006fd0 tx=0x7fed44006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 msgr2=0x7fed50195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50195bc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fed480062a0 tx=0x7fed4800bd40 comp rx=0 tx=0).stop 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 shutdown_connections 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fed380387c0 0x7fed3803ac80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 --2- 172.21.15.196:0/1791462514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed50100720 0x7fed50195bc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 >> 172.21.15.196:0/1791462514 conn(0x7fed500fbcc0 msgr2=0x7fed500fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 shutdown_connections 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:45.164+0000 7fed573b9700 1 -- 172.21.15.196:0/1791462514 wait complete. 2024-03-26T22:05:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:45.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:45 smithi194 ceph-mon[29693]: pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:46.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:46 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1791462514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:46.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:46.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:46.960 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:47.006 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:47.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:47 smithi194 ceph-mon[29693]: pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 -- 172.21.15.196:0/2017275676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 msgr2=0x7fbc38103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 --2- 172.21.15.196:0/2017275676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38103590 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009910 tx=0x7fbc20009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 -- 172.21.15.196:0/2017275676 shutdown_connections 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 --2- 172.21.15.196:0/2017275676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38103590 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 -- 172.21.15.196:0/2017275676 >> 172.21.15.196:0/2017275676 conn(0x7fbc380fabb0 msgr2=0x7fbc380fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 -- 172.21.15.196:0/2017275676 shutdown_connections 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.523+0000 7fbc3e30f700 1 -- 172.21.15.196:0/2017275676 wait complete. 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc3e30f700 1 Processor -- start 2024-03-26T22:05:48.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc3e30f700 1 -- start start 2024-03-26T22:05:48.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc3e30f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:48.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc3e30f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc381960c0 con 0x7fbc381011a0 2024-03-26T22:05:48.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc37fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:48.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc37fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60080/0 (socket says 172.21.15.196:60080) 2024-03-26T22:05:48.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc37fff700 1 -- 172.21.15.196:0/1784443394 learned_addr learned my addr 172.21.15.196:0/1784443394 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:48.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.524+0000 7fbc37fff700 1 -- 172.21.15.196:0/1784443394 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200095f0 con 0x7fbc381011a0 2024-03-26T22:05:48.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.525+0000 7fbc37fff700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38195b80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009ed0 tx=0x7fbc20004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:48.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.525+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc2001d070 con 0x7fbc381011a0 2024-03-26T22:05:48.526 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.525+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc381962c0 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.525+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc2000be20 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.525+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc2000f460 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.525+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc38196760 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.526+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fbc2000f650 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.526+0000 7fbc357fa700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbc240387d0 0x7fbc2403ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.526+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3818f540 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.526+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbc2004cbf0 con 0x7fbc381011a0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.526+0000 7fbc377fe700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbc240387d0 0x7fbc2403ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:48.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.527+0000 7fbc377fe700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbc240387d0 0x7fbc2403ac90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc28006fd0 tx=0x7fbc28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:48.531 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.531+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc2002c440 con 0x7fbc381011a0 2024-03-26T22:05:48.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.800+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc380623e0 con 0x7fbc381011a0 2024-03-26T22:05:48.801 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.801+0000 7fbc357fa700 1 -- 172.21.15.196:0/1784443394 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbc20020b60 con 0x7fbc381011a0 2024-03-26T22:05:48.801 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:48.801 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbc240387d0 msgr2=0x7fbc2403ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbc240387d0 0x7fbc2403ac90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc28006fd0 tx=0x7fbc28006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 msgr2=0x7fbc38195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38195b80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009ed0 tx=0x7fbc20004690 comp rx=0 tx=0).stop 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 shutdown_connections 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fbc240387d0 0x7fbc2403ac90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 --2- 172.21.15.196:0/1784443394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc381011a0 0x7fbc38195b80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:48.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 >> 172.21.15.196:0/1784443394 conn(0x7fbc380fabb0 msgr2=0x7fbc380fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:48.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 shutdown_connections 2024-03-26T22:05:48.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:48.803+0000 7fbc3e30f700 1 -- 172.21.15.196:0/1784443394 wait complete. 2024-03-26T22:05:48.804 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:50.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:49 smithi194 ceph-mon[29693]: pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:50.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:49 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1784443394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:50.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:50.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:50.583 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:50.629 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:52.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:51 smithi194 ceph-mon[29693]: pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:52.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1723288913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 msgr2=0x7f6e881035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 --2- 172.21.15.196:0/1723288913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e881035e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000b0a0 tx=0x7f6e8000b3b0 comp rx=0 tx=0).stop 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1723288913 shutdown_connections 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 --2- 172.21.15.196:0/1723288913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e881035e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1723288913 >> 172.21.15.196:0/1723288913 conn(0x7f6e880fac20 msgr2=0x7f6e880fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1723288913 shutdown_connections 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.206+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1723288913 wait complete. 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.207+0000 7f6e8cad7700 1 Processor -- start 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.207+0000 7f6e8cad7700 1 -- start start 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.207+0000 7f6e8cad7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e88195c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.207+0000 7f6e8cad7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e88196150 con 0x7f6e881011f0 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.207+0000 7f6e8659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e88195c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.207+0000 7f6e8659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e88195c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:60098/0 (socket says 172.21.15.196:60098) 2024-03-26T22:05:52.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e8659c700 1 -- 172.21.15.196:0/1737017174 learned_addr learned my addr 172.21.15.196:0/1737017174 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:52.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e8659c700 1 -- 172.21.15.196:0/1737017174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80009d00 con 0x7f6e881011f0 2024-03-26T22:05:52.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e8659c700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e88195c10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e800062a0 tx=0x7f6e8000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8000e040 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e80003cd0 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e88196350 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e80012450 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.208+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e881967f0 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.209+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8818f590 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.209+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f6e80004040 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.209+0000 7f6e7f7fe700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6e740387b0 0x7f6e7403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.209+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6e8004c670 con 0x7f6e881011f0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.209+0000 7f6e85d9b700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6e740387b0 0x7f6e7403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.210+0000 7f6e85d9b700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6e740387b0 0x7f6e7403ac70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e70006fd0 tx=0x7f6e70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:52.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.215+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e80019310 con 0x7f6e881011f0 2024-03-26T22:05:52.481 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.482+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e880623e0 con 0x7f6e881011f0 2024-03-26T22:05:52.481 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.483+0000 7f6e7f7fe700 1 -- 172.21.15.196:0/1737017174 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6e880623e0 con 0x7f6e881011f0 2024-03-26T22:05:52.482 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:52.482 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:52.483 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.484+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6e740387b0 msgr2=0x7f6e7403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.484+0000 7f6e8cad7700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6e740387b0 0x7f6e7403ac70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e70006fd0 tx=0x7f6e70006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.484+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 msgr2=0x7f6e88195c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e88195c10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e800062a0 tx=0x7f6e8000bd40 comp rx=0 tx=0).stop 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 shutdown_connections 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f6e740387b0 0x7f6e7403ac70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 --2- 172.21.15.196:0/1737017174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e881011f0 0x7f6e88195c10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 >> 172.21.15.196:0/1737017174 conn(0x7f6e880fac20 msgr2=0x7f6e880fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 shutdown_connections 2024-03-26T22:05:52.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:52.485+0000 7f6e8cad7700 1 -- 172.21.15.196:0/1737017174 wait complete. 2024-03-26T22:05:52.485 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:53.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:52 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1737017174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:54.124 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:54.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:54.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:53 smithi194 ceph-mon[29693]: pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:54.326 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:54.372 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:55.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:55 smithi194 ceph-mon[29693]: pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 -- 172.21.15.196:0/2964412375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 msgr2=0x7f26100ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 --2- 172.21.15.196:0/2964412375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100ffa60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2604009910 tx=0x7f2604009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 -- 172.21.15.196:0/2964412375 shutdown_connections 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 --2- 172.21.15.196:0/2964412375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100ffa60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 -- 172.21.15.196:0/2964412375 >> 172.21.15.196:0/2964412375 conn(0x7f26100fac00 msgr2=0x7f26100fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 -- 172.21.15.196:0/2964412375 shutdown_connections 2024-03-26T22:05:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.869+0000 7f26150d4700 1 -- 172.21.15.196:0/2964412375 wait complete. 2024-03-26T22:05:55.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.870+0000 7f26150d4700 1 Processor -- start 2024-03-26T22:05:55.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.870+0000 7f26150d4700 1 -- start start 2024-03-26T22:05:55.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.870+0000 7f26150d4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:55.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.870+0000 7f26150d4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2610069a20 con 0x7f26100ff640 2024-03-26T22:05:55.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.870+0000 7f260ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:55.871 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f260ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36652/0 (socket says 172.21.15.196:36652) 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f260ed9d700 1 -- 172.21.15.196:0/851990162 learned_addr learned my addr 172.21.15.196:0/851990162 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f260ed9d700 1 -- 172.21.15.196:0/851990162 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26040095f0 con 0x7f26100ff640 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f260ed9d700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100694e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2604006010 tx=0x7f26040050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f260401d070 con 0x7f26100ff640 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f260400be20 con 0x7f26100ff640 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26101988f0 con 0x7f26100ff640 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2604022de0 con 0x7f26100ff640 2024-03-26T22:05:55.872 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.871+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2610198d90 con 0x7f26100ff640 2024-03-26T22:05:55.873 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.872+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f26040226c0 con 0x7f26100ff640 2024-03-26T22:05:55.873 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.872+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261018e470 con 0x7f26100ff640 2024-03-26T22:05:55.873 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.872+0000 7f25f7fff700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f80387b0 0x7f25f803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:55.873 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.873+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f260404cb60 con 0x7f26100ff640 2024-03-26T22:05:55.874 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.873+0000 7f260e59c700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f80387b0 0x7f25f803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:55.874 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.873+0000 7f260e59c700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f80387b0 0x7f25f803ac70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2600006fd0 tx=0x7f2600006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:55.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:55.878+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f260402ba00 con 0x7f26100ff640 2024-03-26T22:05:56.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.145+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26100623e0 con 0x7f26100ff640 2024-03-26T22:05:56.145 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.146+0000 7f25f7fff700 1 -- 172.21.15.196:0/851990162 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f260402b3b0 con 0x7f26100ff640 2024-03-26T22:05:56.146 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:56.146 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f80387b0 msgr2=0x7f25f803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f80387b0 0x7f25f803ac70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2600006fd0 tx=0x7f2600006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 msgr2=0x7f26100694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100694e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2604006010 tx=0x7f26040050b0 comp rx=0 tx=0).stop 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 shutdown_connections 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f25f80387b0 0x7f25f803ac70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 --2- 172.21.15.196:0/851990162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26100ff640 0x7f26100694e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 >> 172.21.15.196:0/851990162 conn(0x7f26100fac00 msgr2=0x7f26100fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 shutdown_connections 2024-03-26T22:05:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:56.148+0000 7f26150d4700 1 -- 172.21.15.196:0/851990162 wait complete. 2024-03-26T22:05:56.149 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:05:56.665 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:56 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/851990162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:05:57.665 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:57 smithi194 ceph-mon[29693]: pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:05:57.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:05:57.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:05:57.915 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:57.960 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:05:59.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 -- 172.21.15.196:0/1819997700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 msgr2=0x7fb1f4100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:59.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 --2- 172.21.15.196:0/1819997700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4100fe0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009910 tx=0x7fb1f0009c20 comp rx=0 tx=0).stop 2024-03-26T22:05:59.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 -- 172.21.15.196:0/1819997700 shutdown_connections 2024-03-26T22:05:59.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 --2- 172.21.15.196:0/1819997700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4100fe0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 -- 172.21.15.196:0/1819997700 >> 172.21.15.196:0/1819997700 conn(0x7fb1f40fa370 msgr2=0x7fb1f40fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 -- 172.21.15.196:0/1819997700 shutdown_connections 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.392+0000 7fb1fbccd700 1 -- 172.21.15.196:0/1819997700 wait complete. 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1fbccd700 1 Processor -- start 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1fbccd700 1 -- start start 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1fbccd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4194b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1fbccd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4195040 con 0x7fb1f4100bc0 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1f9a69700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4194b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1f9a69700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4194b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:36672/0 (socket says 172.21.15.196:36672) 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.393+0000 7fb1f9a69700 1 -- 172.21.15.196:0/3353519773 learned_addr learned my addr 172.21.15.196:0/3353519773 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:05:59.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1f9a69700 1 -- 172.21.15.196:0/3353519773 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00095f0 con 0x7fb1f4100bc0 2024-03-26T22:05:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1f9a69700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4194b00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0006010 tx=0x7fb1f0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f001f070 con 0x7fb1f4100bc0 2024-03-26T22:05:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1f0005dc0 con 0x7fb1f4100bc0 2024-03-26T22:05:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f4195240 con 0x7fb1f4100bc0 2024-03-26T22:05:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0024620 con 0x7fb1f4100bc0 2024-03-26T22:05:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.394+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f41956e0 con 0x7fb1f4100bc0 2024-03-26T22:05:59.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.395+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f40623e0 con 0x7fb1f4100bc0 2024-03-26T22:05:59.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.395+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb1f0019650 con 0x7fb1f4100bc0 2024-03-26T22:05:59.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.395+0000 7fb1eaffd700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1e00387c0 0x7fb1e003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:05:59.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.395+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb1f004eb00 con 0x7fb1f4100bc0 2024-03-26T22:05:59.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.395+0000 7fb1f9268700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1e00387c0 0x7fb1e003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:05:59.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.396+0000 7fb1f9268700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1e00387c0 0x7fb1e003ac80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4006fd0 tx=0x7fb1e4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:05:59.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.400+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1f0050e60 con 0x7fb1f4100bc0 2024-03-26T22:05:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.670+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1f4105660 con 0x7fb1f4100bc0 2024-03-26T22:05:59.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.671+0000 7fb1eaffd700 1 -- 172.21.15.196:0/3353519773 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb1f0029070 con 0x7fb1f4100bc0 2024-03-26T22:05:59.671 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:05:59.671 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:05:59.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.672+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1e00387c0 msgr2=0x7fb1e003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:59.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.672+0000 7fb1fbccd700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1e00387c0 0x7fb1e003ac80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4006fd0 tx=0x7fb1e4006e40 comp rx=0 tx=0).stop 2024-03-26T22:05:59.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 msgr2=0x7fb1f4194b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4194b00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0006010 tx=0x7fb1f0009f80 comp rx=0 tx=0).stop 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 shutdown_connections 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb1e00387c0 0x7fb1e003ac80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 --2- 172.21.15.196:0/3353519773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f4100bc0 0x7fb1f4194b00 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 >> 172.21.15.196:0/3353519773 conn(0x7fb1f40fa370 msgr2=0x7fb1f40fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 shutdown_connections 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:05:59.673+0000 7fb1fbccd700 1 -- 172.21.15.196:0/3353519773 wait complete. 2024-03-26T22:05:59.673 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:00.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:59 smithi194 ceph-mon[29693]: pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:00.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:05:59 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3353519773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:01.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:06:01.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:06:01.404 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:01.449 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:02.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:01 smithi194 ceph-mon[29693]: pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:02.930 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/4236256541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 msgr2=0x7f1d18102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 --2- 172.21.15.196:0/4236256541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18102130 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009910 tx=0x7f1d08009c20 comp rx=0 tx=0).stop 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/4236256541 shutdown_connections 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 --2- 172.21.15.196:0/4236256541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18102130 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/4236256541 >> 172.21.15.196:0/4236256541 conn(0x7f1d180fa310 msgr2=0x7f1d180fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/4236256541 shutdown_connections 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.930+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/4236256541 wait complete. 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d1d7bf700 1 Processor -- start 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d1d7bf700 1 -- start start 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d1d7bf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:02.931 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d1d7bf700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d18196110 con 0x7f1d18101d10 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d16ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d16ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:34994/0 (socket says 172.21.15.196:34994) 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d16ffd700 1 -- 172.21.15.196:0/1161145912 learned_addr learned my addr 172.21.15.196:0/1161145912 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.931+0000 7f1d16ffd700 1 -- 172.21.15.196:0/1161145912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d080095f0 con 0x7f1d18101d10 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.932+0000 7f1d16ffd700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18195bd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1d08006010 tx=0x7f1d08009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.932+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d0801d070 con 0x7f1d18101d10 2024-03-26T22:06:02.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.932+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d18196310 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.932+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d08005dc0 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.932+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d08022e90 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.932+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d181967b0 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.933+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1818f560 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.933+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f1d0800b960 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.933+0000 7f1d14ff9700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1d04038760 0x7f1d0403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.933+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1d0804c800 con 0x7f1d18101d10 2024-03-26T22:06:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.933+0000 7f1d0e7fc700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1d04038760 0x7f1d0403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:02.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.934+0000 7f1d0e7fc700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1d04038760 0x7f1d0403ac20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d00006fd0 tx=0x7f1d00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:02.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:02.938+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1d08022750 con 0x7f1d18101d10 2024-03-26T22:06:03.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.207+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d180623e0 con 0x7f1d18101d10 2024-03-26T22:06:03.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.208+0000 7f1d14ff9700 1 -- 172.21.15.196:0/1161145912 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1d180623e0 con 0x7f1d18101d10 2024-03-26T22:06:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:06:03.208 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:06:03.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1d04038760 msgr2=0x7f1d0403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:03.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1d04038760 0x7f1d0403ac20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d00006fd0 tx=0x7f1d00006e40 comp rx=0 tx=0).stop 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 msgr2=0x7f1d18195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18195bd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1d08006010 tx=0x7f1d08009f80 comp rx=0 tx=0).stop 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 shutdown_connections 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f1d04038760 0x7f1d0403ac20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 --2- 172.21.15.196:0/1161145912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1d18101d10 0x7f1d18195bd0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 >> 172.21.15.196:0/1161145912 conn(0x7f1d180fa310 msgr2=0x7f1d180fae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 shutdown_connections 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:03.210+0000 7f1d1d7bf700 1 -- 172.21.15.196:0/1161145912 wait complete. 2024-03-26T22:06:03.210 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:04.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:03 smithi194 ceph-mon[29693]: pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:04.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:03 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/1161145912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:04.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:06:04.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:06:05.067 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:05.114 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:05.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:05 smithi194 ceph-mon[29693]: pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 -- 172.21.15.196:0/1974043487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 msgr2=0x7fa4140ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 --2- 172.21.15.196:0/1974043487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140ffa70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa408009910 tx=0x7fa408009c20 comp rx=0 tx=0).stop 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 -- 172.21.15.196:0/1974043487 shutdown_connections 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 --2- 172.21.15.196:0/1974043487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140ffa70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 -- 172.21.15.196:0/1974043487 >> 172.21.15.196:0/1974043487 conn(0x7fa4140fabf0 msgr2=0x7fa4140fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 -- 172.21.15.196:0/1974043487 shutdown_connections 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 -- 172.21.15.196:0/1974043487 wait complete. 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 Processor -- start 2024-03-26T22:06:06.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.598+0000 7fa41aa8c700 1 -- start start 2024-03-26T22:06:06.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa41aa8c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140fca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:06.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa41aa8c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4140fcfa0 con 0x7fa4140ff650 2024-03-26T22:06:06.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa418828700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140fca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:06.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa418828700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140fca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:35014/0 (socket says 172.21.15.196:35014) 2024-03-26T22:06:06.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa418828700 1 -- 172.21.15.196:0/2381488129 learned_addr learned my addr 172.21.15.196:0/2381488129 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:06:06.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa418828700 1 -- 172.21.15.196:0/2381488129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080095f0 con 0x7fa4140ff650 2024-03-26T22:06:06.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.599+0000 7fa418828700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140fca60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa408006010 tx=0x7fa4080050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:06.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.600+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa40801d070 con 0x7fa4140ff650 2024-03-26T22:06:06.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.600+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4140fb1e0 con 0x7fa4140ff650 2024-03-26T22:06:06.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.600+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa40800be20 con 0x7fa4140ff650 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.600+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa408022e50 con 0x7fa4140ff650 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.600+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4140fb680 con 0x7fa4140ff650 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.601+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa414105db0 con 0x7fa4140ff650 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.601+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fa408022700 con 0x7fa4140ff650 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.601+0000 7fa411ffb700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa404038760 0x7fa40403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.601+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa40804cb70 con 0x7fa4140ff650 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.601+0000 7fa413fff700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa404038760 0x7fa40403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.602+0000 7fa413fff700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa404038760 0x7fa40403ac20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa400006fd0 tx=0x7fa400006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:06.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.606+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa40800b960 con 0x7fa4140ff650 2024-03-26T22:06:06.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.878+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4140623e0 con 0x7fa4140ff650 2024-03-26T22:06:06.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.878+0000 7fa411ffb700 1 -- 172.21.15.196:0/2381488129 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa408027020 con 0x7fa4140ff650 2024-03-26T22:06:06.879 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:06:06.879 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:06:06.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.880+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa404038760 msgr2=0x7fa40403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:06.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa404038760 0x7fa40403ac20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa400006fd0 tx=0x7fa400006e40 comp rx=0 tx=0).stop 2024-03-26T22:06:06.880 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 msgr2=0x7fa4140fca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140fca60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa408006010 tx=0x7fa4080050b0 comp rx=0 tx=0).stop 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 shutdown_connections 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fa404038760 0x7fa40403ac20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 --2- 172.21.15.196:0/2381488129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4140ff650 0x7fa4140fca60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 >> 172.21.15.196:0/2381488129 conn(0x7fa4140fabf0 msgr2=0x7fa414068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 shutdown_connections 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:06.881+0000 7fa41aa8c700 1 -- 172.21.15.196:0/2381488129 wait complete. 2024-03-26T22:06:06.881 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:08.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:07 smithi194 ceph-mon[29693]: pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:08.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:07 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/2381488129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:08.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:06:08.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:06:08.763 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:08.809 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:10.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:09 smithi194 ceph-mon[29693]: pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 -- 172.21.15.196:0/652662992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 msgr2=0x7f3050100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 --2- 172.21.15.196:0/652662992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050100ad0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f304800b0a0 tx=0x7f304800b3b0 comp rx=0 tx=0).stop 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 -- 172.21.15.196:0/652662992 shutdown_connections 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 --2- 172.21.15.196:0/652662992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050100ad0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 -- 172.21.15.196:0/652662992 >> 172.21.15.196:0/652662992 conn(0x7f30500fbc70 msgr2=0x7f30500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 -- 172.21.15.196:0/652662992 shutdown_connections 2024-03-26T22:06:10.308 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.306+0000 7f30576b6700 1 -- 172.21.15.196:0/652662992 wait complete. 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f30576b6700 1 Processor -- start 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f30576b6700 1 -- start start 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f30576b6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f30576b6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3050074f60 con 0x7f30501006b0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f3055452700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f3055452700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:35032/0 (socket says 172.21.15.196:35032) 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f3055452700 1 -- 172.21.15.196:0/3599683958 learned_addr learned my addr 172.21.15.196:0/3599683958 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.307+0000 7f3055452700 1 -- 172.21.15.196:0/3599683958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048009d00 con 0x7f30501006b0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.308+0000 7f3055452700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050074a20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f30480062a0 tx=0x7f304800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.308+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f304800e040 con 0x7f30501006b0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.308+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3048003d70 con 0x7f30501006b0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.308+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30500731a0 con 0x7f30501006b0 2024-03-26T22:06:10.309 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.308+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f304801c890 con 0x7f30501006b0 2024-03-26T22:06:10.310 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.308+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3050073640 con 0x7f30501006b0 2024-03-26T22:06:10.310 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.309+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f304801cb50 con 0x7f30501006b0 2024-03-26T22:06:10.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.309+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3050106e10 con 0x7f30501006b0 2024-03-26T22:06:10.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.309+0000 7f304e7fc700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f30380387c0 0x7f303803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:10.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.309+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f304804c8e0 con 0x7f30501006b0 2024-03-26T22:06:10.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.309+0000 7f3054c51700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f30380387c0 0x7f303803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:10.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.309+0000 7f3054c51700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f30380387c0 0x7f303803ac80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3044006fd0 tx=0x7f3044006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:10.314 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.314+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f304801a590 con 0x7f30501006b0 2024-03-26T22:06:10.584 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.584+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30500623e0 con 0x7f30501006b0 2024-03-26T22:06:10.584 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.584+0000 7f304e7fc700 1 -- 172.21.15.196:0/3599683958 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f304801a3b0 con 0x7f30501006b0 2024-03-26T22:06:10.584 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:06:10.585 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:06:10.586 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f30380387c0 msgr2=0x7f303803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:10.586 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f30380387c0 0x7f303803ac80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3044006fd0 tx=0x7f3044006e40 comp rx=0 tx=0).stop 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 msgr2=0x7f3050074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050074a20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f30480062a0 tx=0x7f304800bd40 comp rx=0 tx=0).stop 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 shutdown_connections 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f30380387c0 0x7f303803ac80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 --2- 172.21.15.196:0/3599683958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30501006b0 0x7f3050074a20 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.586+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 >> 172.21.15.196:0/3599683958 conn(0x7f30500fbc70 msgr2=0x7f30500fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.587+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 shutdown_connections 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:10.587+0000 7f30576b6700 1 -- 172.21.15.196:0/3599683958 wait complete. 2024-03-26T22:06:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:11.066 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:10 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3599683958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:12.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:06:12.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:06:12.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:11 smithi194 ceph-mon[29693]: pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:12.276 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:12.322 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.773+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3256979813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 msgr2=0x7f3ac01035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 --2- 172.21.15.196:0/3256979813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac01035b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0009910 tx=0x7f3ab0009c20 comp rx=0 tx=0).stop 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3256979813 shutdown_connections 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 --2- 172.21.15.196:0/3256979813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac01035b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3256979813 >> 172.21.15.196:0/3256979813 conn(0x7f3ac00fabd0 msgr2=0x7f3ac00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3256979813 shutdown_connections 2024-03-26T22:06:13.774 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3256979813 wait complete. 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 Processor -- start 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.774+0000 7f3ac7b84700 1 -- start start 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac7b84700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac0195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac7b84700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac01960e0 con 0x7f3ac01011c0 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac5920700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac0195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac5920700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac0195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57804/0 (socket says 172.21.15.196:57804) 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac5920700 1 -- 172.21.15.196:0/3544430869 learned_addr learned my addr 172.21.15.196:0/3544430869 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac5920700 1 -- 172.21.15.196:0/3544430869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab00095f0 con 0x7f3ac01011c0 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac5920700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac0195ba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0006010 tx=0x7f3ab00050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:13.777 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab001d070 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ab000be20 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.775+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac01962e0 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.776+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab000f460 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.776+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac0196780 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.776+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7f3ab0022470 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.776+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac018f560 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.777+0000 7f3ab6ffd700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3aac0387d0 0x7f3aac03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.777+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ab004ccd0 con 0x7f3ac01011c0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.777+0000 7f3ac511f700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3aac0387d0 0x7f3aac03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.777+0000 7f3ac511f700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3aac0387d0 0x7f3aac03ac90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006fd0 tx=0x7f3abc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:13.781 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:13.782+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3ab0005c00 con 0x7f3ac01011c0 2024-03-26T22:06:14.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.052+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ac00623e0 con 0x7f3ac01011c0 2024-03-26T22:06:14.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.052+0000 7f3ab6ffd700 1 -- 172.21.15.196:0/3544430869 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3ac00623e0 con 0x7f3ac01011c0 2024-03-26T22:06:14.053 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:06:14.053 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:06:14.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.054+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3aac0387d0 msgr2=0x7f3aac03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:14.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.054+0000 7f3ac7b84700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3aac0387d0 0x7f3aac03ac90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006fd0 tx=0x7f3abc006e40 comp rx=0 tx=0).stop 2024-03-26T22:06:14.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.054+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 msgr2=0x7f3ac0195ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.054+0000 7f3ac7b84700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac0195ba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0006010 tx=0x7f3ab00050b0 comp rx=0 tx=0).stop 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.055+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 shutdown_connections 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.055+0000 7f3ac7b84700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7f3aac0387d0 0x7f3aac03ac90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.055+0000 7f3ac7b84700 1 --2- 172.21.15.196:0/3544430869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac01011c0 0x7f3ac0195ba0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.055+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 >> 172.21.15.196:0/3544430869 conn(0x7f3ac00fabd0 msgr2=0x7f3ac00fb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.055+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 shutdown_connections 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:14.055+0000 7f3ac7b84700 1 -- 172.21.15.196:0/3544430869 wait complete. 2024-03-26T22:06:14.055 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:14.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:13 smithi194 ceph-mon[29693]: pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:15.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:14 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3544430869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:15.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:06:15.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:06:15.959 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:16.005 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:16.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:15 smithi194 ceph-mon[29693]: pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:17.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 -- 172.21.15.196:0/3101543562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 msgr2=0x7fb7480ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:17.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 --2- 172.21.15.196:0/3101543562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb7480ffa40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb738009910 tx=0x7fb738009c20 comp rx=0 tx=0).stop 2024-03-26T22:06:17.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 -- 172.21.15.196:0/3101543562 shutdown_connections 2024-03-26T22:06:17.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 --2- 172.21.15.196:0/3101543562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb7480ffa40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:17.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 -- 172.21.15.196:0/3101543562 >> 172.21.15.196:0/3101543562 conn(0x7fb7480fabc0 msgr2=0x7fb7480fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:17.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 -- 172.21.15.196:0/3101543562 shutdown_connections 2024-03-26T22:06:17.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.476+0000 7fb74d686700 1 -- 172.21.15.196:0/3101543562 wait complete. 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb74d686700 1 Processor -- start 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb74d686700 1 -- start start 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb74d686700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb748194a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb74d686700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb748194fd0 con 0x7fb7480ff620 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb746ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb748194a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb746ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb748194a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57816/0 (socket says 172.21.15.196:57816) 2024-03-26T22:06:17.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb746ffd700 1 -- 172.21.15.196:0/234440882 learned_addr learned my addr 172.21.15.196:0/234440882 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.477+0000 7fb746ffd700 1 -- 172.21.15.196:0/234440882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380095f0 con 0x7fb7480ff620 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.478+0000 7fb746ffd700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb748194a90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb738006010 tx=0x7fb738009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.478+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb73801d070 con 0x7fb7480ff620 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.478+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb738005dc0 con 0x7fb7480ff620 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.478+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7481951d0 con 0x7fb7480ff620 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.478+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb738022e90 con 0x7fb7480ff620 2024-03-26T22:06:17.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.478+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb748195670 con 0x7fb7480ff620 2024-03-26T22:06:17.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.479+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74818e450 con 0x7fb7480ff620 2024-03-26T22:06:17.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.479+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb738022470 con 0x7fb7480ff620 2024-03-26T22:06:17.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.479+0000 7fb72ffff700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb730038760 0x7fb73003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:17.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.479+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb73804c7f0 con 0x7fb7480ff620 2024-03-26T22:06:17.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.479+0000 7fb7467fc700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb730038760 0x7fb73003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:17.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.480+0000 7fb7467fc700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb730038760 0x7fb73003ac20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb73c006fd0 tx=0x7fb73c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:17.484 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.484+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb73800f4f0 con 0x7fb7480ff620 2024-03-26T22:06:17.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.751+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7480623e0 con 0x7fb7480ff620 2024-03-26T22:06:17.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.752+0000 7fb72ffff700 1 -- 172.21.15.196:0/234440882 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb7480623e0 con 0x7fb7480ff620 2024-03-26T22:06:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:06:17.752 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb730038760 msgr2=0x7fb73003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb730038760 0x7fb73003ac20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb73c006fd0 tx=0x7fb73c006e40 comp rx=0 tx=0).stop 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 msgr2=0x7fb748194a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb748194a90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb738006010 tx=0x7fb738009f80 comp rx=0 tx=0).stop 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 shutdown_connections 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb730038760 0x7fb73003ac20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 --2- 172.21.15.196:0/234440882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7480ff620 0x7fb748194a90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:17.754 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 >> 172.21.15.196:0/234440882 conn(0x7fb7480fabc0 msgr2=0x7fb7480fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:17.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 shutdown_connections 2024-03-26T22:06:17.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:17.754+0000 7fb74d686700 1 -- 172.21.15.196:0/234440882 wait complete. 2024-03-26T22:06:17.755 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:18.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:17 smithi194 ceph-mon[29693]: pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:18.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:17 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/234440882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:19.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T22:06:19.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T22:06:19.581 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:19.627 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T22:06:20.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:19 smithi194 ceph-mon[29693]: pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 -- 172.21.15.196:0/3287108986 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 msgr2=0x7fb294100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 --2- 172.21.15.196:0/3287108986 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294100b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb288009910 tx=0x7fb288009c20 comp rx=0 tx=0).stop 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 -- 172.21.15.196:0/3287108986 shutdown_connections 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 --2- 172.21.15.196:0/3287108986 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294100b40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 -- 172.21.15.196:0/3287108986 >> 172.21.15.196:0/3287108986 conn(0x7fb2940fbcc0 msgr2=0x7fb2940fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 -- 172.21.15.196:0/3287108986 shutdown_connections 2024-03-26T22:06:21.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.213+0000 7fb299176700 1 -- 172.21.15.196:0/3287108986 wait complete. 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.214+0000 7fb299176700 1 Processor -- start 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.214+0000 7fb299176700 1 -- start start 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb299176700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb299176700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294196100 con 0x7fb294100720 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb292d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb292d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.196:57830/0 (socket says 172.21.15.196:57830) 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb292d9d700 1 -- 172.21.15.196:0/3031981466 learned_addr learned my addr 172.21.15.196:0/3031981466 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb292d9d700 1 -- 172.21.15.196:0/3031981466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2880095f0 con 0x7fb294100720 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb292d9d700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294195bc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb288006010 tx=0x7fb288009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.215+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28801d070 con 0x7fb294100720 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.216+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb288005dc0 con 0x7fb294100720 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.216+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb294196300 con 0x7fb294100720 2024-03-26T22:06:21.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.216+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb288022e90 con 0x7fb294100720 2024-03-26T22:06:21.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.216+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2941967a0 con 0x7fb294100720 2024-03-26T22:06:21.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.216+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45605+0+0 (secure 0 0 0) 0x7fb28800f4f0 con 0x7fb294100720 2024-03-26T22:06:21.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.217+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29418f550 con 0x7fb294100720 2024-03-26T22:06:21.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.217+0000 7fb27bfff700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb27c038760 0x7fb27c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T22:06:21.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.217+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb28804c7d0 con 0x7fb294100720 2024-03-26T22:06:21.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.217+0000 7fb29259c700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb27c038760 0x7fb27c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T22:06:21.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.217+0000 7fb29259c700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb27c038760 0x7fb27c03ac20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb284006fd0 tx=0x7fb284006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T22:06:21.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.222+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb288017440 con 0x7fb294100720 2024-03-26T22:06:21.488 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.488+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2940623e0 con 0x7fb294100720 2024-03-26T22:06:21.489 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.489+0000 7fb27bfff700 1 -- 172.21.15.196:0/3031981466 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb2940623e0 con 0x7fb294100720 2024-03-26T22:06:21.489 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-26T22:06:21.489 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"2f9b9224-ebbb-11ee-95d0-87774f69a715","modified":"2024-03-26T21:53:45.808507Z","created":"2024-03-26T21:53:45.808507Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi194","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T22:06:21.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb27c038760 msgr2=0x7fb27c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:21.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb27c038760 0x7fb27c03ac20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb284006fd0 tx=0x7fb284006e40 comp rx=0 tx=0).stop 2024-03-26T22:06:21.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 msgr2=0x7fb294195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T22:06:21.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294195bc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb288006010 tx=0x7fb288009f80 comp rx=0 tx=0).stop 2024-03-26T22:06:21.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 shutdown_connections 2024-03-26T22:06:21.491 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:6800/3128608440,v1:172.21.15.194:6801/3128608440] conn(0x7fb27c038760 0x7fb27c03ac20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:21.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 --2- 172.21.15.196:0/3031981466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294100720 0x7fb294195bc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T22:06:21.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 >> 172.21.15.196:0/3031981466 conn(0x7fb2940fbcc0 msgr2=0x7fb2940fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T22:06:21.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 shutdown_connections 2024-03-26T22:06:21.492 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-26T22:06:21.491+0000 7fb299176700 1 -- 172.21.15.196:0/3031981466 wait complete. 2024-03-26T22:06:21.492 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-26T22:06:22.004 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 964, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-03-26T22:06:22.006 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T22:06:22.006 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T22:06:22.027 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T22:06:22.041 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T22:06:22.042 INFO:tasks.cephadm.mon.smithi194:Stopping mon.smithi194... 2024-03-26T22:06:22.042 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194 2024-03-26T22:06:22.096 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:21 smithi194 ceph-mon[29693]: pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T22:06:22.097 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:21 smithi194 ceph-mon[29693]: from='client.? 172.21.15.196:0/3031981466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T22:06:22.415 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:22 smithi194 systemd[1]: Stopping Ceph mon.smithi194 for 2f9b9224-ebbb-11ee-95d0-87774f69a715... 2024-03-26T22:06:22.824 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:22 smithi194 ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194[29689]: 2024-03-26T22:06:22.449+0000 7fd5e44d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi194 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T22:06:22.824 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:22 smithi194 ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194[29689]: 2024-03-26T22:06:22.449+0000 7fd5e44d8700 -1 mon.smithi194@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T22:06:22.824 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:22 smithi194 podman[41305]: 2024-03-26 22:06:22.542463807 +0000 UTC m=+0.279732242 container died f276715a9ec198666af36d2136c0ec3b32b5e44ff8fd349b66d1c6b877280239 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, RELEASE=reef-d83b58f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T22:06:23.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:22 smithi194 podman[41305]: 2024-03-26 22:06:22.982275909 +0000 UTC m=+0.719544385 container cleanup f276715a9ec198666af36d2136c0ec3b32b5e44ff8fd349b66d1c6b877280239 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-d83b58f, org.label-schema.build-date=20240319) 2024-03-26T22:06:23.165 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:Mar 26 22:06:22 smithi194 bash[41305]: ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715-mon-smithi194 2024-03-26T22:06:23.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi194.service' 2024-03-26T22:06:23.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T22:06:23.489 INFO:tasks.cephadm.mon.smithi194:Stopped mon.smithi194 2024-03-26T22:06:23.489 INFO:tasks.cephadm.mon.smithi196:Stopping mon.smithi196... 2024-03-26T22:06:23.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196 2024-03-26T22:06:23.525 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196.service: Unit ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196.service not loaded. 2024-03-26T22:06:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-03-26T22:06:23.527 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi196" Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 838, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 964, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 857, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 5: 'sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196' 2024-03-26T22:06:23.528 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T22:06:23.530 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/remote/smithi194/crash 2024-03-26T22:06:23.531 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/crash -- . 2024-03-26T22:06:23.570 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/remote/smithi196/crash 2024-03-26T22:06:23.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/crash -- . 2024-03-26T22:06:23.618 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T22:06:23.618 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-03-26T22:06:23.619 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T22:06:23.620 DEBUG:teuthology.orchestra.run.smithi194:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-26T22:06:23.652 INFO:tasks.cephadm:Compressing logs... 2024-03-26T22:06:23.652 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T22:06:23.695 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T22:06:23.721 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-mon.smithi194.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi194.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-mon.smithi194.log: /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi194.log: /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr: 92.3% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-client.ceph-exporter.smithi194.log.gz 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr: 79.7% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr: 89.2% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-mgr.smithi194.wndbij.log 2024-03-26T22:06:23.722 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-mgr.smithi194.wndbij.log: 89.3% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T22:06:23.723 INFO:teuthology.orchestra.run.smithi194.stderr: 90.8% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T22:06:23.724 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T22:06:23.724 INFO:teuthology.orchestra.run.smithi196.stderr:: No such file or directory 2024-03-26T22:06:23.726 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/cephadm.log: 96.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T22:06:23.727 INFO:teuthology.orchestra.run.smithi194.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T22:06:23.728 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-26T22:06:23.728 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m0.017s 2024-03-26T22:06:23.728 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m0.011s 2024-03-26T22:06:23.728 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.019s 2024-03-26T22:06:23.766 INFO:teuthology.orchestra.run.smithi194.stderr: 91.6% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-mgr.smithi194.wndbij.log.gz 2024-03-26T22:06:23.893 INFO:teuthology.orchestra.run.smithi194.stderr: 90.6% -- replaced with /var/log/ceph/2f9b9224-ebbb-11ee-95d0-87774f69a715/ceph-mon.smithi194.log.gz 2024-03-26T22:06:23.894 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-26T22:06:23.895 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m0.188s 2024-03-26T22:06:23.895 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m0.235s 2024-03-26T22:06:23.895 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.025s 2024-03-26T22:06:23.895 INFO:tasks.cephadm:Archiving logs... 2024-03-26T22:06:23.895 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/remote/smithi194/log 2024-03-26T22:06:23.896 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T22:06:24.015 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/remote/smithi196/log 2024-03-26T22:06:24.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T22:06:24.048 INFO:tasks.cephadm:Removing cluster... 2024-03-26T22:06:24.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 --force 2024-03-26T22:10:38.440 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T22:10:46.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2f9b9224-ebbb-11ee-95d0-87774f69a715 --force 2024-03-26T22:10:46.340 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 2f9b9224-ebbb-11ee-95d0-87774f69a715 2024-03-26T22:10:46.606 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T22:10:46.606 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T22:10:46.622 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T22:10:46.638 INFO:tasks.cephadm:Teardown complete 2024-03-26T22:10:46.639 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 838, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 964, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 2194, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 1823, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 857, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 5: 'sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196' 2024-03-26T22:10:46.857 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=982c56716ff144fd8ff04f2fd2049197 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 838, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 964, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 2194, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 1823, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks/cephadm.py", line 857, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 5: 'sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196' 2024-03-26T22:10:46.862 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T22:10:46.872 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T22:10:46.880 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T22:10:46.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T22:10:46.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T22:10:46.972 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T22:10:46.980 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T22:10:46.981 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T22:10:46.983 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T22:10:46.996 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-26T22:10:46.997 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-26T22:10:47.000 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T22:10:47.000 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-26T22:10:47.001 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +462us[ +479us] +/- 78ms 2024-03-26T22:10:47.001 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -898us[ -881us] +/- 58ms 2024-03-26T22:10:47.001 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +720us[ +720us] +/- 78ms 2024-03-26T22:10:47.001 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T22:10:47.002 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T22:10:47.003 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-26T22:10:47.003 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +468us[ +473us] +/- 78ms 2024-03-26T22:10:47.003 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -874us[ -869us] +/- 58ms 2024-03-26T22:10:47.003 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +752us[ +757us] +/- 78ms 2024-03-26T22:10:47.003 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T22:10:47.003 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T22:10:47.013 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T22:10:47.013 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T22:10:47.022 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T22:10:47.111 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T22:10:47.184 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T22:10:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:10:47.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T22:10:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:10:47.244 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T22:10:47.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T22:10:47.262 INFO:teuthology.task.internal:Duration was 1481.215675 seconds 2024-03-26T22:10:47.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T22:10:47.271 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T22:10:47.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T22:10:47.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T22:10:47.314 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T22:10:47.327 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T22:10:47.700 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T22:10:47.700 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-03-26T22:10:47.700 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T22:10:47.759 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-26T22:10:47.759 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T22:10:47.784 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T22:10:47.784 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T22:10:47.802 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T22:10:47.853 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T22:10:47.854 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T22:10:47.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T22:10:48.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T22:10:48.269 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T22:10:48.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T22:10:48.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T22:10:48.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T22:10:48.333 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:10:48.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:10:48.363 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-03-26T22:10:48.388 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-26T22:10:48.432 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:10:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:10:48.447 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:10:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:10:48.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T22:10:48.497 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T22:10:48.498 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/remote/smithi194 2024-03-26T22:10:48.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T22:10:48.537 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/remote/smithi196 2024-03-26T22:10:48.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T22:10:48.579 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T22:10:48.579 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T22:10:48.582 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T22:10:48.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T22:10:48.638 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T22:10:48.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T22:10:48.648 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T22:10:48.648 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T22:10:48.651 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T22:10:48.663 INFO:teuthology.orchestra.run.smithi194.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 22:10 /home/ubuntu/cephtest 2024-03-26T22:10:48.681 INFO:teuthology.orchestra.run.smithi196.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 22:10 /home/ubuntu/cephtest 2024-03-26T22:10:48.682 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T22:10:48.693 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T22:10:48.715 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T22:10:48.736 DEBUG:teuthology.nuke:shortname: smithi194 2024-03-26T22:10:48.737 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T22:10:48.756 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-26T22:10:48.756 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T22:10:48.760 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:36:13.192283', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuJlGUFZj0dqznIhEZ+MceqG2b/P6HvERcGC9PK+u7wG0xCR7x5k53UR19rNTkBH3/aNqe5B4Eu8pSjuNQzyv4='} 2024-03-26T22:10:48.780 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:36:13.193501', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH1ye03rR2irGWQ5NmnAkqEhkvtxrNZ/v8LYn1Nrd71KYevo/lz2SNIUvLg5SRSG/Q3dLxVTM5IhWiYgIUK/ilQ='} 2024-03-26T22:10:48.809 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-26T22:10:48.810 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T22:10:48.833 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-26T22:10:48.833 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:10:48.847 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-26T22:10:48.847 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T22:10:48.855 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-26T22:10:48.869 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-26T22:10:48.869 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:10:48.893 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-26T22:10:52.857 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:10:52.894 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:10:52.968 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-26T22:10:53.004 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-26T22:10:56.973 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:10:57.005 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:10:57.085 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-26T22:10:57.116 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-26T22:10:57.186 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-26T22:10:57.217 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-26T22:10:57.321 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1481.2156751155853 failure_reason: 'Command failed on smithi196 with status 5: ''sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196''' flavor: default owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=982c56716ff144fd8ff04f2fd2049197 status: fail success: false 2024-03-26T22:10:57.322 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T22:10:57.430 INFO:teuthology.run:FAIL