2024-02-20T13:02:06.237 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-20T13:02:06.237 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709 -- /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/orig.config.yaml 2024-02-20T13:02:06.288 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T13:02:06.376 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709 branch: wip-yuri5-testing-2024-02-16-1120-quincy description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7567709' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-02-16-1120-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2024-02-16-1120-quincy sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4250 sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri5-testing-2024-02-16-1120-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQLIMGqyei2iIGo1IJY4+EnIdvWofaXj/gZE0rzlnawmU5fH6mPaRsy3FVFAsVUI5nf5KM68XoxSZ0jdlO9EHs= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOl5YEEZOed9gZqRtxggC/V12jeAgn4UTiu1Ju7iQCdDRqwJjcyRFk7QyIF2U2IKM8A1RS5s7kPBZHIUxLcHLc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_20:21:18 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-20T13:02:06.376 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa; will attempt to use it 2024-02-20T13:02:06.377 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks 2024-02-20T13:02:06.377 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-20T13:02:06.378 INFO:teuthology.task.internal:Checking packages... 2024-02-20T13:02:06.402 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b' 2024-02-20T13:02:06.403 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:02:06.403 INFO:teuthology.packaging:ref: None 2024-02-20T13:02:06.403 INFO:teuthology.packaging:tag: None 2024-02-20T13:02:06.403 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-02-16-1120-quincy 2024-02-20T13:02:06.403 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:02:06.403 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2024-02-16-1120-quincy 2024-02-20T13:02:06.658 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-543.gb0bd01b8 2024-02-20T13:02:06.698 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-20T13:02:06.705 INFO:teuthology.task.internal:no buildpackages task found 2024-02-20T13:02:06.705 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-20T13:02:06.720 INFO:teuthology.task.internal:Saving configuration 2024-02-20T13:02:06.738 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-20T13:02:06.775 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T13:02:06.798 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 12:57:04.711169', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQLIMGqyei2iIGo1IJY4+EnIdvWofaXj/gZE0rzlnawmU5fH6mPaRsy3FVFAsVUI5nf5KM68XoxSZ0jdlO9EHs='} 2024-02-20T13:02:06.819 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 12:57:04.713052', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOl5YEEZOed9gZqRtxggC/V12jeAgn4UTiu1Ju7iQCdDRqwJjcyRFk7QyIF2U2IKM8A1RS5s7kPBZHIUxLcHLc='} 2024-02-20T13:02:06.819 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-20T13:02:06.825 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-20T13:02:06.825 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-20T13:02:06.825 INFO:teuthology.run_tasks:Running task console_log... 2024-02-20T13:02:06.897 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f472242b160>, signals=[15]) 2024-02-20T13:02:06.898 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-20T13:02:06.904 INFO:teuthology.task.internal:Opening connections... 2024-02-20T13:02:06.904 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-02-20T13:02:06.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:02:06.982 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-02-20T13:02:06.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:02:07.053 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-20T13:02:07.060 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-02-20T13:02:07.083 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-02-20T13:02:07.084 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-02-20T13:02:07.145 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T13:02:07.146 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-20T13:02:07.146 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T13:02:07.146 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-02-20T13:02:07.172 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-02-20T13:02:07.194 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-02-20T13:02:07.195 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-20T13:02:07.255 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-02-20T13:02:07.256 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T13:02:07.256 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-20T13:02:07.256 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T13:02:07.256 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-02-20T13:02:07.280 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-20T13:02:07.289 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-20T13:02:07.294 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-20T13:02:07.294 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-02-20T13:02:07.300 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-02-20T13:02:07.318 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-20T13:02:07.324 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-20T13:02:07.325 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-02-20T13:02:07.358 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-02-20T13:02:07.390 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-20T13:02:07.444 INFO:teuthology.run_tasks:Running task kernel... 2024-02-20T13:02:07.457 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-20T13:02:07.458 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-20T13:02:07.458 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T13:02:07.458 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T13:02:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:02:07.479 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-02-20T13:02:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:02:07.480 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-02-20T13:02:07.540 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-489.el8.x86_64 2024-02-20T13:02:07.540 INFO:teuthology.task.kernel:Running kernel on smithi114: 4.18.0-489.el8.x86_64 2024-02-20T13:02:07.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-02-20T13:02:07.542 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-489.el8.x86_64 2024-02-20T13:02:07.542 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.18.0-489.el8.x86_64 2024-02-20T13:02:07.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-02-20T13:02:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 7.1 MB/s | 27 MB 00:03 2024-02-20T13:02:14.078 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - AppStream 6.8 MB/s | 27 MB 00:03 2024-02-20T13:02:24.123 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - BaseOS 2.0 MB/s | 10 MB 00:05 2024-02-20T13:02:24.673 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 1.8 MB/s | 10 MB 00:05 2024-02-20T13:02:26.270 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-20T13:02:26.687 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-20T13:02:26.907 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-20T13:02:26.922 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-20T13:02:27.233 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 6.9 kB 00:00 2024-02-20T13:02:27.651 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-02-20T13:02:27.991 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2024-02-20T13:02:28.810 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-02-20T13:02:31.731 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 376 kB/s | 24 kB 00:00 2024-02-20T13:02:32.794 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 410 kB/s | 24 kB 00:00 2024-02-20T13:02:34.695 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-20T13:02:34.695 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-20T13:02:34.799 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:02:34.800 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:02:34.800 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-20T13:02:34.800 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout:Install 3 Packages 2024-02-20T13:02:34.801 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:02:34.802 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 90 M 2024-02-20T13:02:34.802 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 97 M 2024-02-20T13:02:34.802 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:02:35.933 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-20T13:02:35.933 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-20T13:02:36.047 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:02:36.048 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:02:36.048 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-02-20T13:02:36.048 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:02:36.048 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:02:36.048 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-20T13:02:36.048 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout:Install 3 Packages 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:02:36.049 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 90 M 2024-02-20T13:02:36.050 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 97 M 2024-02-20T13:02:36.050 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:02:37.257 INFO:teuthology.orchestra.run.smithi114.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 4.6 MB/s | 10 MB 00:02 2024-02-20T13:02:38.237 INFO:teuthology.orchestra.run.smithi038.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 5.3 MB/s | 10 MB 00:01 2024-02-20T13:02:38.608 INFO:teuthology.orchestra.run.smithi114.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-02-20T13:02:39.466 INFO:teuthology.orchestra.run.smithi114.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 9.7 MB/s | 43 MB 00:04 2024-02-20T13:02:39.466 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:02:39.467 INFO:teuthology.orchestra.run.smithi114.stdout:Total 19 MB/s | 90 MB 00:04 2024-02-20T13:02:39.470 INFO:teuthology.orchestra.run.smithi038.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2024-02-20T13:02:39.987 INFO:teuthology.orchestra.run.smithi038.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 12 MB/s | 43 MB 00:03 2024-02-20T13:02:39.987 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:02:39.988 INFO:teuthology.orchestra.run.smithi038.stdout:Total 23 MB/s | 90 MB 00:03 2024-02-20T13:02:40.152 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:02:40.586 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:02:40.586 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:02:40.700 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:02:41.142 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:02:41.143 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:02:41.818 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:02:41.844 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:02:42.430 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:02:42.450 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:02:44.585 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:02:45.067 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:02:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:02:45.896 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:02:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:02:49.092 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:02:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:02:50.087 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:02:52.529 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:02:52.814 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:02:53.764 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:02:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:15.058 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:15.081 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:15.423 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:15.423 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:03:15.423 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:03:16.018 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:16.018 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:03:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:03:16.018 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-20T13:03:16.018 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-20T13:03:16.018 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:03:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:03:16.173 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:03:16.389 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:03:16.389 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:03:16.389 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-20T13:03:16.389 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T13:03:16.389 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T13:03:16.390 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-20T13:03:16.390 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T13:03:16.390 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:03:16.599 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:03:16.600 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:03:16.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-02-20T13:03:16.633 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-02-20T13:03:16.633 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-02-20T13:03:16.633 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.02 2024-02-20T13:03:16.633 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 129.el8 2024-02-20T13:03:16.633 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-02-20T13:03:16.633 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Group : System Environment/Base 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 9265586 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Relocations : (not relocatable) 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : CentOS BuildSystem 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:03:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T13:03:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T13:03:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T13:03:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-02-20T13:03:16.635 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:03:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T13:03:16.636 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T13:03:16.636 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 4.18.0-539.el8.x86_64 2024-02-20T13:03:16.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T13:03:17.309 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:17.335 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:17.702 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:17.703 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-20T13:03:17.703 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-20T13:03:17.930 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-02-20T13:03:18.168 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T13:03:18.168 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:03:18.168 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:03:18.168 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-20T13:03:18.168 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-20T13:03:18.168 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:03:18.169 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:03:18.334 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:03:18.557 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:03:18.557 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:03:18.557 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-20T13:03:18.557 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T13:03:18.557 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T13:03:18.558 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-20T13:03:18.558 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T13:03:18.558 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T13:03:18.777 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T13:03:18.778 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T13:03:18.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.02 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 129.el8 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Group : System Environment/Base 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 9265586 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-02-20T13:03:18.812 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Relocations : (not relocatable) 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : CentOS BuildSystem 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T13:03:18.813 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-02-20T13:03:18.814 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:03:18.814 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T13:03:18.814 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T13:03:18.815 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 4.18.0-539.el8.x86_64 2024-02-20T13:03:18.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T13:03:20.372 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-02-20T13:03:20.482 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-02-20T13:03:20.483 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-02-20T13:03:20.516 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.ljsLd0DTad 2024-02-20T13:03:20.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ljsLd0DTad 2024-02-20T13:03:20.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.ljsLd0DTad 2024-02-20T13:03:20.735 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.ljsLd0DTad is 6KB 2024-02-20T13:03:20.782 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.ljsLd0DTad 2024-02-20T13:03:20.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T13:03:20.866 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-20T13:03:20.866 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-20T13:03:20.866 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-20T13:03:20.866 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-20T13:03:20.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-20T13:03:21.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-02-20T13:03:22.843 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-02-20T13:03:22.844 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-02-20T13:03:22.872 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.5lDOhFanTs 2024-02-20T13:03:22.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5lDOhFanTs 2024-02-20T13:03:22.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.5lDOhFanTs 2024-02-20T13:03:23.091 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.5lDOhFanTs is 6KB 2024-02-20T13:03:23.141 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.5lDOhFanTs 2024-02-20T13:03:23.158 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T13:03:23.230 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-20T13:03:23.230 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-20T13:03:23.230 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-20T13:03:23.231 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-20T13:03:23.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-20T13:03:23.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-02-20T13:03:51.068 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-20T13:03:51.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:03:51.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:03:53.374 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-20T13:03:53.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-02-20T13:03:53.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:04:09.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-02-20T13:04:11.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-02-20T13:04:18.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:04:18.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:04:20.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-02-20T13:04:20.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:04:21.596 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-02-20T13:04:23.904 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-02-20T13:04:33.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:04:33.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:04:35.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-02-20T13:04:35.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:04:52.028 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-02-20T13:05:07.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:05:07.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:05:07.725 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-02-20T13:05:07.994 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-02-20T13:05:07.994 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-20T13:05:07.995 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-02-20T13:05:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-539.el8.x86_64 2024-02-20T13:05:08.012 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-20T13:05:08.012 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T13:05:08.012 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T13:05:08.064 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-02-20T13:05:09.013 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-20T13:05:09.014 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T13:05:09.117 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-02-20T13:05:09.133 DEBUG:teuthology.parallel:result is None 2024-02-20T13:05:09.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:05:09.327 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-20T13:05:09.327 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-02-20T13:05:09.353 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-539.el8.x86_64 2024-02-20T13:05:09.353 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-20T13:05:09.353 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T13:05:09.353 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T13:05:10.354 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-20T13:05:10.354 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T13:05:10.437 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-02-20T13:05:10.454 DEBUG:teuthology.parallel:result is None 2024-02-20T13:05:10.454 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-20T13:05:10.462 INFO:teuthology.task.internal:Creating test directory... 2024-02-20T13:05:10.462 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T13:05:10.496 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T13:05:10.659 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-20T13:05:10.667 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-20T13:05:10.674 INFO:teuthology.task.internal:Creating archive directory... 2024-02-20T13:05:10.674 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T13:05:10.723 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T13:05:10.961 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-20T13:05:10.967 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-20T13:05:10.967 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T13:05:11.007 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T13:05:11.041 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:05:11.051 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:05:11.537 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:05:11.636 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T13:05:11.640 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-20T13:05:11.648 INFO:teuthology.task.internal:Configuring sudo... 2024-02-20T13:05:11.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T13:05:11.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T13:05:11.962 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-20T13:05:11.971 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-20T13:05:11.971 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T13:05:11.973 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T13:05:11.994 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:05:12.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:05:12.226 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:05:12.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:05:12.314 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:05:12.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T13:05:12.381 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:05:12.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T13:05:12.480 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:05:12.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T13:05:12.611 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:05:12.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T13:05:12.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-02-20T13:05:12.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-02-20T13:05:12.739 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:05:12.789 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:05:13.137 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-20T13:05:13.146 INFO:teuthology.task.internal:Starting timer... 2024-02-20T13:05:13.147 INFO:teuthology.run_tasks:Running task pcp... 2024-02-20T13:05:13.160 INFO:teuthology.run_tasks:Running task selinux... 2024-02-20T13:05:13.171 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-20T13:05:13.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-02-20T13:05:13.250 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-02-20T13:05:13.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-02-20T13:05:13.319 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-02-20T13:05:13.321 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-20T13:05:13.322 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-02-20T13:05:13.349 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-02-20T13:05:13.349 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-02-20T13:05:13.374 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-02-20T13:05:13.374 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-02-20T13:05:13.374 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:05:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:05:13.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:05:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:05:13.443 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-20T13:05:13.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-02-20T13:05:13.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-02-20T13:05:13.518 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-20T13:05:13.528 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-20T13:05:13.546 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-20T13:05:13.565 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-20T13:11:42.244 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-02-20T13:11:42.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-02-20T13:11:42.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:11:42.323 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-02-20T13:11:42.395 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-02-20T13:11:42.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:11:42.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T13:11:42.469 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-02-20T13:11:42.539 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-20T13:11:42.540 INFO:teuthology.run_tasks:Running task clock... 2024-02-20T13:11:42.551 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-20T13:11:42.551 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T13:11:42.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:11:42.554 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T13:11:42.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:11:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T13:11:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T13:11:42.613 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T13:11:42.627 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T13:11:42.671 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-02-20T13:11:42.682 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-02-20T13:11:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-02-20T13:11:42.697 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T13:11:42.701 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-02-20T13:11:42.710 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T13:11:42.715 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T13:11:42.728 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T13:11:42.776 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2024-02-20T13:11:42.779 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:11:42.779 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-02-20T13:11:42.779 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.779 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.779 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.780 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.791 INFO:teuthology.orchestra.run.smithi114.stderr:bash: ntpq: command not found 2024-02-20T13:11:42.795 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:11:42.795 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-02-20T13:11:42.795 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.795 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.795 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.795 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:11:42.796 INFO:teuthology.run_tasks:Running task pexec... 2024-02-20T13:11:42.805 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-20T13:11:42.806 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi038.front.sepia.ceph.com 2024-02-20T13:11:42.806 DEBUG:teuthology.orchestra.run.smithi038:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T13:11:42.806 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2024-02-20T13:11:42.806 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T13:11:43.619 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:04 ago on Tue 20 Feb 2024 01:09:39 PM UTC. 2024-02-20T13:11:43.642 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 01:09:49 PM UTC. 2024-02-20T13:11:45.662 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:11:45.663 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:11:45.663 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-20T13:11:45.663 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:11:45.664 INFO:teuthology.orchestra.run.smithi114.stdout:Resetting modules: 2024-02-20T13:11:45.664 INFO:teuthology.orchestra.run.smithi114.stdout: container-tools 2024-02-20T13:11:45.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:45.664 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:11:45.664 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:11:45.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:45.833 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout:Resetting modules: 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:11:45.834 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:11:45.835 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:45.915 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:11:46.138 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:11:46.781 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:07 ago on Tue 20 Feb 2024 01:09:39 PM UTC. 2024-02-20T13:11:47.024 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:57 ago on Tue 20 Feb 2024 01:09:49 PM UTC. 2024-02-20T13:11:48.816 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout:Installing group/module packages: 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T13:11:48.851 INFO:teuthology.orchestra.run.smithi114.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout:Installing module profiles: 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: container-tools/common 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:48.852 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling module streams: 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout: container-tools rhel8 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:48.853 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 28 M 2024-02-20T13:11:48.854 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 92 M 2024-02-20T13:11:48.854 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:11:49.198 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout:Installing group/module packages: 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T13:11:49.238 INFO:teuthology.orchestra.run.smithi038.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout:Installing module profiles: 2024-02-20T13:11:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools/common 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling module streams: 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools rhel8 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-02-20T13:11:49.240 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:49.241 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 28 M 2024-02-20T13:11:49.241 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 92 M 2024-02-20T13:11:49.241 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:11:49.420 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.8 MB/s | 688 kB 00:00 2024-02-20T13:11:49.487 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 569 kB/s | 38 kB 00:00 2024-02-20T13:11:49.562 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.9 MB/s | 1.0 MB 00:00 2024-02-20T13:11:49.638 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 1.7 MB/s | 257 kB 00:00 2024-02-20T13:11:49.713 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.4 MB/s | 180 kB 00:00 2024-02-20T13:11:49.780 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 369 kB/s | 25 kB 00:00 2024-02-20T13:11:49.847 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-02-20T13:11:49.861 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.6 MB/s | 688 kB 00:00 2024-02-20T13:11:49.927 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.0 MB/s | 1.0 MB 00:00 2024-02-20T13:11:49.952 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 416 kB/s | 38 kB 00:00 2024-02-20T13:11:50.027 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 2.5 MB/s | 257 kB 00:00 2024-02-20T13:11:50.094 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.6 MB/s | 180 kB 00:00 2024-02-20T13:11:50.161 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 372 kB/s | 25 kB 00:00 2024-02-20T13:11:50.228 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-02-20T13:11:50.372 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 5.3 MB/s | 4.3 MB 00:00 2024-02-20T13:11:50.706 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 5.8 MB/s | 9.7 MB 00:01 2024-02-20T13:11:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 6.7 MB/s | 2.5 MB 00:00 2024-02-20T13:11:50.773 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 788 kB/s | 53 kB 00:00 2024-02-20T13:11:50.815 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 613 kB/s | 41 kB 00:00 2024-02-20T13:11:50.948 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 4.3 MB/s | 4.3 MB 00:00 2024-02-20T13:11:51.082 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 7.1 MB/s | 8.8 MB 00:01 2024-02-20T13:11:51.082 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:11:51.082 INFO:teuthology.orchestra.run.smithi114.stdout:Total 12 MB/s | 28 MB 00:02 2024-02-20T13:11:51.196 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 5.5 MB/s | 9.7 MB 00:01 2024-02-20T13:11:51.262 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 792 kB/s | 53 kB 00:00 2024-02-20T13:11:51.351 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:11:51.354 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): toolbox-0.0.99.5-1.module_el8+847+7863 6.2 MB/s | 2.5 MB 00:00 2024-02-20T13:11:51.371 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 377 kB/s | 41 kB 00:00 2024-02-20T13:11:51.372 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:11:51.372 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:11:51.522 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 6.8 MB/s | 8.8 MB 00:01 2024-02-20T13:11:51.522 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:11:51.522 INFO:teuthology.orchestra.run.smithi038.stdout:Total 12 MB/s | 28 MB 00:02 2024-02-20T13:11:51.622 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:11:51.625 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:11:51.801 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:11:51.823 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:11:51.823 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:11:52.089 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:11:52.092 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:11:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:11:52.826 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:11:53.326 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T13:11:53.529 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:11:53.707 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T13:11:53.802 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T13:11:53.914 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T13:11:53.990 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:11:54.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T13:11:54.192 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T13:11:54.381 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T13:11:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T13:11:54.753 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T13:11:55.231 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T13:11:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T13:11:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T13:11:56.188 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T13:11:56.251 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T13:11:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T13:11:56.606 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T13:11:56.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:11:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T13:11:57.041 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:11:57.223 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:11:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T13:11:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:11:57.643 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:11:57.643 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T13:11:57.643 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:11:57.644 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:11:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:11:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T13:11:58.124 INFO:teuthology.orchestra.run.smithi114.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:11:58.125 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:11:58.423 DEBUG:teuthology.parallel:result is None 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T13:11:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T13:11:58.455 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T13:11:58.455 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T13:11:59.024 INFO:teuthology.orchestra.run.smithi038.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:11:59.025 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:11:59.270 DEBUG:teuthology.parallel:result is None 2024-02-20T13:11:59.271 INFO:teuthology.run_tasks:Running task install... 2024-02-20T13:11:59.283 DEBUG:teuthology.task.install:project ceph 2024-02-20T13:11:59.283 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b'}} 2024-02-20T13:11:59.283 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b'} 2024-02-20T13:11:59.283 INFO:teuthology.task.install:Using flavor: default 2024-02-20T13:11:59.292 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-20T13:11:59.292 INFO:teuthology.task.install:extra packages: [] 2024-02-20T13:11:59.292 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b', 'tag': None, 'wait_for_package': False} 2024-02-20T13:11:59.292 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:11:59.293 INFO:teuthology.packaging:ref: None 2024-02-20T13:11:59.293 INFO:teuthology.packaging:tag: None 2024-02-20T13:11:59.293 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:11:59.293 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:11:59.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:11:59.295 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b', 'tag': None, 'wait_for_package': False} 2024-02-20T13:11:59.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:11:59.296 INFO:teuthology.packaging:ref: None 2024-02-20T13:11:59.296 INFO:teuthology.packaging:tag: None 2024-02-20T13:11:59.296 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:11:59.296 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:11:59.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:11:59.548 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/ 2024-02-20T13:11:59.548 INFO:teuthology.task.install.rpm:Package version is 16.2.14-548.g81bd20d6 2024-02-20T13:11:59.609 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/ 2024-02-20T13:11:59.610 INFO:teuthology.task.install.rpm:Package version is 16.2.14-548.g81bd20d6 2024-02-20T13:11:59.676 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T13:11:59.677 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:11:59.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T13:11:59.711 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T13:11:59.711 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:11:59.711 INFO:teuthology.packaging:ref: None 2024-02-20T13:11:59.711 INFO:teuthology.packaging:tag: None 2024-02-20T13:11:59.711 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:11:59.711 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:11:59.711 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T13:11:59.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T13:11:59.797 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T13:11:59.797 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:11:59.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T13:11:59.831 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T13:11:59.831 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:11:59.831 INFO:teuthology.packaging:ref: None 2024-02-20T13:11:59.831 INFO:teuthology.packaging:tag: None 2024-02-20T13:11:59.831 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:11:59.831 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:11:59.832 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T13:11:59.863 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T13:11:59.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T13:11:59.931 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-02-20T13:11:59.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-02-20T13:11:59.986 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T13:12:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-02-20T13:12:00.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-02-20T13:12:00.206 INFO:teuthology.orchestra.run.smithi114.stdout:76 files removed 2024-02-20T13:12:00.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-02-20T13:12:00.337 INFO:teuthology.orchestra.run.smithi038.stdout:76 files removed 2024-02-20T13:12:00.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-02-20T13:12:00.999 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 210 kB/s | 76 kB 00:00 2024-02-20T13:12:01.157 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 211 kB/s | 76 kB 00:00 2024-02-20T13:12:01.308 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 56 kB/s | 15 kB 00:00 2024-02-20T13:12:01.458 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 59 kB/s | 15 kB 00:00 2024-02-20T13:12:01.650 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2024-02-20T13:12:01.835 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2024-02-20T13:12:03.759 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - AppStream 13 MB/s | 27 MB 00:02 2024-02-20T13:12:04.401 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 11 MB/s | 27 MB 00:02 2024-02-20T13:12:14.052 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - BaseOS 1.9 MB/s | 10 MB 00:05 2024-02-20T13:12:14.810 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 2.0 MB/s | 10 MB 00:05 2024-02-20T13:12:16.219 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-20T13:12:16.628 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-20T13:12:17.036 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 34 kB/s | 18 kB 00:00 2024-02-20T13:12:17.453 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-20T13:12:17.604 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - PowerTools 5.5 MB/s | 4.8 MB 00:00 2024-02-20T13:12:18.271 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - PowerTools 6.7 MB/s | 4.8 MB 00:00 2024-02-20T13:12:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-20T13:12:19.397 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-20T13:12:20.832 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-02-20T13:12:23.009 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 16 MB 00:04 2024-02-20T13:12:24.799 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 395 kB/s | 24 kB 00:00 2024-02-20T13:12:26.823 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 400 kB/s | 24 kB 00:00 2024-02-20T13:12:28.683 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:12:28.686 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:12:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:12:28.686 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:12:28.686 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:12:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 12 M 2024-02-20T13:12:28.686 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 4.2 M 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.1 M 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 24 M 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 26 k 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 825 k 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T13:12:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 494 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.8 M 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 45 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 99 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 215 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 389 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 369 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 114 k 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:12:28.688 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:12:28.689 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:12:28.689 INFO:teuthology.orchestra.run.smithi038.stdout:Install 21 Packages 2024-02-20T13:12:28.689 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-02-20T13:12:28.689 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:12:28.689 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 58 M 2024-02-20T13:12:28.689 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:12:30.353 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 12 M 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 4.2 M 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.1 M 2024-02-20T13:12:30.356 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 24 M 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 26 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 825 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 494 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.8 M 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T13:12:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 45 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 99 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 215 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 389 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 369 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 114 k 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout:Install 21 Packages 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-02-20T13:12:30.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:12:30.359 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 58 M 2024-02-20T13:12:30.359 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:12:30.550 INFO:teuthology.orchestra.run.smithi038.stdout:(1/23): ceph-radosgw-16.2.14-548.g81bd20d6.el8. 9.1 MB/s | 12 MB 00:01 2024-02-20T13:12:30.592 INFO:teuthology.orchestra.run.smithi038.stdout:(2/23): ceph-selinux-16.2.14-548.g81bd20d6.el8. 619 kB/s | 26 kB 00:00 2024-02-20T13:12:30.725 INFO:teuthology.orchestra.run.smithi038.stdout:(3/23): libcephfs2-16.2.14-548.g81bd20d6.el8.x8 6.0 MB/s | 825 kB 00:00 2024-02-20T13:12:30.809 INFO:teuthology.orchestra.run.smithi038.stdout:(4/23): ceph-base-16.2.14-548.g81bd20d6.el8.x86 4.0 MB/s | 6.1 MB 00:01 2024-02-20T13:12:30.859 INFO:teuthology.orchestra.run.smithi038.stdout:(5/23): libradosstriper1-16.2.14-548.g81bd20d6. 3.6 MB/s | 494 kB 00:00 2024-02-20T13:12:30.909 INFO:teuthology.orchestra.run.smithi038.stdout:(6/23): python3-ceph-argparse-16.2.14-548.g81bd 903 kB/s | 45 kB 00:00 2024-02-20T13:12:31.001 INFO:teuthology.orchestra.run.smithi038.stdout:(7/23): python3-ceph-common-16.2.14-548.g81bd20 1.1 MB/s | 99 kB 00:00 2024-02-20T13:12:31.102 INFO:teuthology.orchestra.run.smithi038.stdout:(8/23): python3-cephfs-16.2.14-548.g81bd20d6.el 2.1 MB/s | 215 kB 00:00 2024-02-20T13:12:31.363 INFO:teuthology.orchestra.run.smithi038.stdout:(9/23): ceph-common-16.2.14-548.g81bd20d6.el8.x 12 MB/s | 24 MB 00:02 2024-02-20T13:12:31.394 INFO:teuthology.orchestra.run.smithi038.stdout:(10/23): python3-rados-16.2.14-548.g81bd20d6.el 1.3 MB/s | 389 kB 00:00 2024-02-20T13:12:31.419 INFO:teuthology.orchestra.run.smithi038.stdout:(11/23): python3-rbd-16.2.14-548.g81bd20d6.el8. 6.5 MB/s | 369 kB 00:00 2024-02-20T13:12:31.445 INFO:teuthology.orchestra.run.smithi038.stdout:(12/23): python3-rgw-16.2.14-548.g81bd20d6.el8. 2.2 MB/s | 114 kB 00:00 2024-02-20T13:12:31.520 INFO:teuthology.orchestra.run.smithi038.stdout:(13/23): librgw2-16.2.14-548.g81bd20d6.el8.x86_ 5.3 MB/s | 3.8 MB 00:00 2024-02-20T13:12:31.754 INFO:teuthology.orchestra.run.smithi038.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 842 kB/s | 259 kB 00:00 2024-02-20T13:12:31.779 INFO:teuthology.orchestra.run.smithi038.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 170 kB/s | 44 kB 00:00 2024-02-20T13:12:31.913 INFO:teuthology.orchestra.run.smithi038.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2024-02-20T13:12:31.938 INFO:teuthology.orchestra.run.smithi038.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-02-20T13:12:31.997 INFO:teuthology.orchestra.run.smithi038.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-02-20T13:12:32.055 INFO:teuthology.orchestra.run.smithi038.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2024-02-20T13:12:32.074 INFO:teuthology.orchestra.run.smithi114.stdout:(1/23): ceph-radosgw-16.2.14-548.g81bd20d6.el8. 8.8 MB/s | 12 MB 00:01 2024-02-20T13:12:32.093 INFO:teuthology.orchestra.run.smithi038.stdout:(20/23): leveldb-1.22-1.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2024-02-20T13:12:32.114 INFO:teuthology.orchestra.run.smithi038.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 344 kB/s | 59 kB 00:00 2024-02-20T13:12:32.116 INFO:teuthology.orchestra.run.smithi114.stdout:(2/23): ceph-selinux-16.2.14-548.g81bd20d6.el8. 618 kB/s | 26 kB 00:00 2024-02-20T13:12:32.192 INFO:teuthology.orchestra.run.smithi114.stdout:(3/23): ceph-base-16.2.14-548.g81bd20d6.el8.x86 4.3 MB/s | 6.1 MB 00:01 2024-02-20T13:12:32.242 INFO:teuthology.orchestra.run.smithi114.stdout:(4/23): libcephfs2-16.2.14-548.g81bd20d6.el8.x8 6.4 MB/s | 825 kB 00:00 2024-02-20T13:12:32.293 INFO:teuthology.orchestra.run.smithi114.stdout:(5/23): libradosstriper1-16.2.14-548.g81bd20d6. 4.8 MB/s | 494 kB 00:00 2024-02-20T13:12:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:(6/23): python3-ceph-argparse-16.2.14-548.g81bd 541 kB/s | 45 kB 00:00 2024-02-20T13:12:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:(7/23): python3-ceph-common-16.2.14-548.g81bd20 986 kB/s | 99 kB 00:00 2024-02-20T13:12:32.515 INFO:teuthology.orchestra.run.smithi038.stdout:(22/23): librbd1-16.2.14-548.g81bd20d6.el8.x86_ 9.8 MB/s | 4.2 MB 00:00 2024-02-20T13:12:32.543 INFO:teuthology.orchestra.run.smithi114.stdout:(8/23): python3-cephfs-16.2.14-548.g81bd20d6.el 3.2 MB/s | 215 kB 00:00 2024-02-20T13:12:32.602 INFO:teuthology.orchestra.run.smithi114.stdout:(9/23): librgw2-16.2.14-548.g81bd20d6.el8.x86_6 11 MB/s | 3.8 MB 00:00 2024-02-20T13:12:32.652 INFO:teuthology.orchestra.run.smithi114.stdout:(10/23): python3-rados-16.2.14-548.g81bd20d6.el 3.5 MB/s | 389 kB 00:00 2024-02-20T13:12:32.677 INFO:teuthology.orchestra.run.smithi114.stdout:(11/23): python3-rbd-16.2.14-548.g81bd20d6.el8. 4.8 MB/s | 369 kB 00:00 2024-02-20T13:12:32.703 INFO:teuthology.orchestra.run.smithi114.stdout:(12/23): python3-rgw-16.2.14-548.g81bd20d6.el8. 2.2 MB/s | 114 kB 00:00 2024-02-20T13:12:32.840 INFO:teuthology.orchestra.run.smithi038.stdout:(23/23): librados2-16.2.14-548.g81bd20d6.el8.x8 4.8 MB/s | 3.7 MB 00:00 2024-02-20T13:12:32.844 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:12:32.845 INFO:teuthology.orchestra.run.smithi038.stdout:Total 14 MB/s | 58 MB 00:04 2024-02-20T13:12:32.890 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:12:32.943 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:12:32.943 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:12:33.011 INFO:teuthology.orchestra.run.smithi114.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 842 kB/s | 259 kB 00:00 2024-02-20T13:12:33.112 INFO:teuthology.orchestra.run.smithi114.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 439 kB/s | 44 kB 00:00 2024-02-20T13:12:33.254 INFO:teuthology.orchestra.run.smithi114.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 332 kB/s | 47 kB 00:00 2024-02-20T13:12:33.339 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:12:33.342 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:12:33.871 INFO:teuthology.orchestra.run.smithi114.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 497 kB/s | 306 kB 00:00 2024-02-20T13:12:34.004 INFO:teuthology.orchestra.run.smithi114.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-02-20T13:12:34.321 INFO:teuthology.orchestra.run.smithi114.stdout:(18/23): ceph-common-16.2.14-548.g81bd20d6.el8. 6.8 MB/s | 24 MB 00:03 2024-02-20T13:12:34.347 INFO:teuthology.orchestra.run.smithi114.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 172 kB/s | 59 kB 00:00 2024-02-20T13:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-20T13:12:34.730 INFO:teuthology.orchestra.run.smithi114.stdout:(20/23): librados2-16.2.14-548.g81bd20d6.el8.x8 9.8 MB/s | 3.7 MB 00:00 2024-02-20T13:12:34.755 INFO:teuthology.orchestra.run.smithi114.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 173 kB/s | 75 kB 00:00 2024-02-20T13:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:12:34.972 INFO:teuthology.orchestra.run.smithi114.stdout:(22/23): librbd1-16.2.14-548.g81bd20d6.el8.x86_ 17 MB/s | 4.2 MB 00:00 2024-02-20T13:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:12:35.373 INFO:teuthology.orchestra.run.smithi114.stdout:(23/23): librdkafka-1.6.1-1.el8.x86_64.rpm 251 kB/s | 677 kB 00:02 2024-02-20T13:12:35.377 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:12:35.377 INFO:teuthology.orchestra.run.smithi114.stdout:Total 12 MB/s | 58 MB 00:05 2024-02-20T13:12:35.419 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:12:35.466 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:12:35.466 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:12:35.739 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:12:35.827 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:12:35.830 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:12:35.919 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:12:36.366 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:12:36.536 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:12:36.629 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:12:36.816 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-20T13:12:36.855 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:12:37.090 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-20T13:12:37.268 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:12:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-20T13:12:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:12:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:12:37.674 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-20T13:12:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-20T13:12:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 11/25 2024-02-20T13:12:37.877 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:12:37.955 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:12:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:12:38.174 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:12:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:12:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:12:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 14/25 2024-02-20T13:12:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:12:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:12:38.743 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:12:38.901 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:12:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-20T13:12:38.986 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-20T13:12:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-20T13:12:39.169 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-20T13:12:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-20T13:12:39.520 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 19/25 2024-02-20T13:12:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-20T13:12:39.835 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:12:39.908 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-20T13:12:40.082 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-20T13:12:40.281 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 11/25 2024-02-20T13:12:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:12:40.630 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:12:40.825 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 14/25 2024-02-20T13:12:40.968 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:12:41.071 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-20T13:12:41.295 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-20T13:12:41.476 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-20T13:12:41.655 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-20T13:12:42.095 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 19/25 2024-02-20T13:12:42.149 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:12:42.436 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:12:43.177 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:12:43.636 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:12:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:12:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:12:44.619 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:12:44.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:12:45.550 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-20T13:12:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:12:45.954 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-20T13:12:46.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:13:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:13:03.954 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-02-20T13:13:03.954 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-02-20T13:13:03.954 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2024-02-20T13:13:03.954 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /run 2024-02-20T13:13:03.954 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-02-20T13:13:03.955 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-02-20T13:13:03.955 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-02-20T13:13:03.955 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-02-20T13:13:03.955 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2024-02-20T13:13:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:05.175 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:13:05.238 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:13:05.239 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /dev 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /run 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-02-20T13:13:05.241 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-02-20T13:13:05.242 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-02-20T13:13:05.242 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /dev 2024-02-20T13:13:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:05.443 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:13:05.443 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:13:06.107 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:13:06.107 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 1/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 6/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 7/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 8/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 9/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 10/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 11/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:13:06.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-20T13:13:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 24/25 2024-02-20T13:13:06.413 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:13:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-20T13:13:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:13:06.719 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-20T13:13:06.719 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:13:07.013 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T13:13:07.014 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:07.016 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:07.016 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:07.370 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:13:07.370 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 1/25 2024-02-20T13:13:07.370 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2/25 2024-02-20T13:13:07.370 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-20T13:13:07.370 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-20T13:13:07.370 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 6/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 7/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 8/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 9/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 10/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 11/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-20T13:13:07.371 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-20T13:13:07.372 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-20T13:13:07.372 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-20T13:13:07.372 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-20T13:13:07.372 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 24/25 2024-02-20T13:13:07.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-02-20T13:13:08.223 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:44 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-20T13:13:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:08.328 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-02-20T13:13:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:08.330 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:13:08.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-02-20T13:13:09.357 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:43 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:13:09.873 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 52 M 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 174 k 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T13:13:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:09.876 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:13:09.876 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:09.876 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-02-20T13:13:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:09.876 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 53 M 2024-02-20T13:13:09.876 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 197 M 2024-02-20T13:13:09.877 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:13:10.262 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): libcephsqlite-16.2.14-548.g81bd20d6.el8. 865 kB/s | 174 kB 00:00 2024-02-20T13:13:10.554 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 412 kB/s | 203 kB 00:00 2024-02-20T13:13:10.704 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 426 kB/s | 188 kB 00:00 2024-02-20T13:13:10.871 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 424 kB/s | 71 kB 00:00 2024-02-20T13:13:10.903 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 52 M 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 174 k 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T13:13:10.904 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 53 M 2024-02-20T13:13:10.905 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 197 M 2024-02-20T13:13:10.906 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:13:11.105 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 587 kB/s | 323 kB 00:00 2024-02-20T13:13:11.537 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 365 kB/s | 203 kB 00:00 2024-02-20T13:13:11.595 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): libcephsqlite-16.2.14-548.g81bd20d6.el8. 283 kB/s | 174 kB 00:00 2024-02-20T13:13:12.004 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 403 kB/s | 188 kB 00:00 2024-02-20T13:13:12.212 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 339 kB/s | 71 kB 00:00 2024-02-20T13:13:12.279 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 472 kB/s | 323 kB 00:00 2024-02-20T13:13:12.571 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): ceph-test-16.2.14-548.g81bd20d6.el8.x86_ 21 MB/s | 52 MB 00:02 2024-02-20T13:13:12.576 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:12.576 INFO:teuthology.orchestra.run.smithi038.stdout:Total 20 MB/s | 53 MB 00:02 2024-02-20T13:13:12.618 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:13:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:13:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:13:12.945 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:13:12.948 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:13:13.474 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:13:13.724 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T13:13:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T13:13:14.056 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:13:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:13:14.305 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T13:13:14.498 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:13:19.406 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:13:19.654 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:13:19.952 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:13:19.952 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 1/6 2024-02-20T13:13:19.952 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2/6 2024-02-20T13:13:19.952 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T13:13:19.952 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T13:13:19.952 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T13:13:20.410 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: jq-1.6-8.el8.x86_64 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:20.411 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:20.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-02-20T13:13:21.447 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:57 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:23.095 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.4 k 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 2.4 M 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 1.6 M 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 245 k 2024-02-20T13:13:23.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 5.2 M 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 19 M 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T13:13:23.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:23.101 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:13:23.102 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:23.102 INFO:teuthology.orchestra.run.smithi038.stdout:Install 32 Packages 2024-02-20T13:13:23.102 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:23.102 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 32 M 2024-02-20T13:13:23.102 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 104 M 2024-02-20T13:13:23.102 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:13:23.540 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-16.2.14-548.g81bd20d6.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2024-02-20T13:13:23.898 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-mds-16.2.14-548.g81bd20d6.el8.x86_ 4.5 MB/s | 2.4 MB 00:00 2024-02-20T13:13:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): ceph-mgr-16.2.14-548.g81bd20d6.el8.x86_ 2.8 MB/s | 1.6 MB 00:00 2024-02-20T13:13:24.049 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): ceph-mgr-modules-core-16.2.14-548.g81bd 2.2 MB/s | 245 kB 00:00 2024-02-20T13:13:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 650 kB/s | 157 kB 00:00 2024-02-20T13:13:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 516 kB/s | 39 kB 00:00 2024-02-20T13:13:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.2 MB/s | 103 kB 00:00 2024-02-20T13:13:24.517 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): ceph-mon-16.2.14-548.g81bd20d6.el8.x86_ 5.3 MB/s | 5.2 MB 00:00 2024-02-20T13:13:24.542 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 622 kB/s | 57 kB 00:00 2024-02-20T13:13:24.759 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2024-02-20T13:13:24.976 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): ceph-osd-16.2.14-548.g81bd20d6.el8.x86 17 MB/s | 19 MB 00:01 2024-02-20T13:13:25.085 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 806 kB/s | 457 kB 00:00 2024-02-20T13:13:25.119 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-20T13:13:25.144 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 653 kB/s | 109 kB 00:00 2024-02-20T13:13:25.436 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 124 kB/s | 44 kB 00:00 2024-02-20T13:13:25.511 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 472 kB/s | 185 kB 00:00 2024-02-20T13:13:25.536 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 440 kB/s | 173 kB 00:00 2024-02-20T13:13:25.612 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-02-20T13:13:25.637 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-02-20T13:13:25.687 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2024-02-20T13:13:25.712 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 485 kB/s | 49 kB 00:00 2024-02-20T13:13:25.754 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 501 kB/s | 59 kB 00:00 2024-02-20T13:13:25.788 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-02-20T13:13:25.805 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2024-02-20T13:13:25.839 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2024-02-20T13:13:25.881 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 260 kB/s | 24 kB 00:00 2024-02-20T13:13:25.897 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 316 kB/s | 29 kB 00:00 2024-02-20T13:13:25.923 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2024-02-20T13:13:25.981 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-02-20T13:13:26.015 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 929 kB/s | 85 kB 00:00 2024-02-20T13:13:26.074 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2024-02-20T13:13:26.099 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-02-20T13:13:26.100 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:26.100 INFO:teuthology.orchestra.run.smithi038.stdout:Total 11 MB/s | 32 MB 00:02 2024-02-20T13:13:26.184 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:13:26.226 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:13:26.227 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:13:26.541 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:13:26.544 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:13:27.070 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): ceph-test-16.2.14-548.g81bd20d6.el8.x86_ 3.2 MB/s | 52 MB 00:16 2024-02-20T13:13:27.072 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:27.072 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.3 MB/s | 53 MB 00:16 2024-02-20T13:13:27.114 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:13:27.135 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:13:27.135 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:13:27.422 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:13:27.425 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:13:27.696 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:13:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T13:13:27.929 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:13:28.129 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T13:13:28.141 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T13:13:28.339 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T13:13:28.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T13:13:28.461 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:13:28.541 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T13:13:28.644 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T13:13:28.743 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T13:13:28.834 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T13:13:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:13:29.016 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T13:13:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T13:13:29.402 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T13:13:29.639 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T13:13:29.868 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T13:13:30.139 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T13:13:30.363 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T13:13:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T13:13:31.041 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T13:13:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T13:13:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T13:13:31.756 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:13:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T13:13:32.328 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T13:13:32.524 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T13:13:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T13:13:32.963 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T13:13:33.176 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T13:13:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T13:13:33.526 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-20T13:13:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T13:13:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:13:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T13:13:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-20T13:13:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 1/6 2024-02-20T13:13:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2/6 2024-02-20T13:13:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T13:13:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T13:13:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T13:13:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 27/32 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-8.el8.x86_64 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:34.563 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:13:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:13:34.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-02-20T13:13:35.539 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:09 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:13:36.264 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:13:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:13:36.911 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:13:37.112 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:13:37.114 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.4 k 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 2.4 M 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 1.6 M 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 245 k 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 5.2 M 2024-02-20T13:13:37.118 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 19 M 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T13:13:37.119 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T13:13:37.120 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:37.121 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:13:37.121 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:37.121 INFO:teuthology.orchestra.run.smithi114.stdout:Install 32 Packages 2024-02-20T13:13:37.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:37.121 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 32 M 2024-02-20T13:13:37.122 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 104 M 2024-02-20T13:13:37.122 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:13:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:13:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:13:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:13:37.647 INFO:teuthology.orchestra.run.smithi114.stdout:(1/32): ceph-16.2.14-548.g81bd20d6.el8.x86_64.r 53 kB/s | 6.4 kB 00:00 2024-02-20T13:13:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:13:38.039 INFO:teuthology.orchestra.run.smithi114.stdout:(2/32): ceph-mon-16.2.14-548.g81bd20d6.el8.x86_ 13 MB/s | 5.2 MB 00:00 2024-02-20T13:13:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 1/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 3/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 4/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 5/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 6/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T13:13:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T13:13:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T13:13:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T13:13:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T13:13:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T13:13:38.264 INFO:teuthology.orchestra.run.smithi114.stdout:(3/32): ceph-mds-16.2.14-548.g81bd20d6.el8.x86_ 3.3 MB/s | 2.4 MB 00:00 2024-02-20T13:13:38.297 INFO:teuthology.orchestra.run.smithi114.stdout:(4/32): ceph-mgr-16.2.14-548.g81bd20d6.el8.x86_ 2.1 MB/s | 1.6 MB 00:00 2024-02-20T13:13:38.348 INFO:teuthology.orchestra.run.smithi114.stdout:(5/32): ceph-mgr-modules-core-16.2.14-548.g81bd 2.9 MB/s | 245 kB 00:00 2024-02-20T13:13:38.723 INFO:teuthology.orchestra.run.smithi114.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 103 kB/s | 39 kB 00:00 2024-02-20T13:13:38.749 INFO:teuthology.orchestra.run.smithi114.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 348 kB/s | 157 kB 00:00 2024-02-20T13:13:38.941 INFO:teuthology.orchestra.run.smithi114.stdout:(8/32): ceph-osd-16.2.14-548.g81bd20d6.el8.x86_ 21 MB/s | 19 MB 00:00 2024-02-20T13:13:38.983 INFO:teuthology.orchestra.run.smithi114.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 399 kB/s | 103 kB 00:00 2024-02-20T13:13:39.058 INFO:teuthology.orchestra.run.smithi114.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 185 kB/s | 57 kB 00:00 2024-02-20T13:13:39.233 INFO:teuthology.orchestra.run.smithi114.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 995 kB/s | 249 kB 00:00 2024-02-20T13:13:39.325 INFO:teuthology.orchestra.run.smithi114.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2024-02-20T13:13:39.384 INFO:teuthology.orchestra.run.smithi114.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-02-20T13:13:39.428 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T13:13:39.428 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:39.428 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T13:13:39.429 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T13:13:39.430 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T13:13:39.431 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T13:13:39.431 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T13:13:39.431 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T13:13:39.431 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T13:13:39.431 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:39.431 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:39.601 INFO:teuthology.orchestra.run.smithi114.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 158 kB/s | 44 kB 00:00 2024-02-20T13:13:39.634 INFO:teuthology.orchestra.run.smithi114.stdout:(15/32): python3-werkzeug-0.12.2-4.el8.noarch.r 660 kB/s | 457 kB 00:00 2024-02-20T13:13:39.735 INFO:teuthology.orchestra.run.smithi114.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 535 kB/s | 185 kB 00:00 2024-02-20T13:13:39.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-02-20T13:13:39.785 INFO:teuthology.orchestra.run.smithi114.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 938 kB/s | 173 kB 00:00 2024-02-20T13:13:39.839 INFO:teuthology.orchestra.run.smithi114.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 107 kB/s | 11 kB 00:00 2024-02-20T13:13:39.869 INFO:teuthology.orchestra.run.smithi114.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2024-02-20T13:13:39.919 INFO:teuthology.orchestra.run.smithi114.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 607 kB/s | 49 kB 00:00 2024-02-20T13:13:39.953 INFO:teuthology.orchestra.run.smithi114.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 702 kB/s | 59 kB 00:00 2024-02-20T13:13:40.028 INFO:teuthology.orchestra.run.smithi114.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2024-02-20T13:13:40.053 INFO:teuthology.orchestra.run.smithi114.stdout:(23/32): python3-cherrypy-18.4.0-1.el8.noarch.r 918 kB/s | 384 kB 00:00 2024-02-20T13:13:40.079 INFO:teuthology.orchestra.run.smithi114.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-02-20T13:13:40.104 INFO:teuthology.orchestra.run.smithi114.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-02-20T13:13:40.129 INFO:teuthology.orchestra.run.smithi114.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-02-20T13:13:40.155 INFO:teuthology.orchestra.run.smithi114.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-02-20T13:13:40.180 INFO:teuthology.orchestra.run.smithi114.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-02-20T13:13:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-02-20T13:13:40.289 INFO:teuthology.orchestra.run.smithi114.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 787 kB/s | 85 kB 00:00 2024-02-20T13:13:40.314 INFO:teuthology.orchestra.run.smithi114.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-20T13:13:40.339 INFO:teuthology.orchestra.run.smithi114.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 271 kB/s | 23 kB 00:00 2024-02-20T13:13:40.343 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:40.343 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.9 MB/s | 32 MB 00:03 2024-02-20T13:13:40.424 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:13:40.463 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:13:40.463 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:13:40.539 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:16 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:40.934 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:13:40.937 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:13:42.088 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:13:42.175 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:13:42.224 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:42.226 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:13:42.226 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:42.323 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T13:13:42.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-02-20T13:13:42.509 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T13:13:42.803 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T13:13:43.074 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T13:13:43.116 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:19 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:43.289 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T13:13:43.461 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T13:13:43.654 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T13:13:43.815 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T13:13:44.042 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T13:13:44.278 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T13:13:44.550 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T13:13:44.783 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 85 k 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:13:44.784 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:44.785 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:13:44.785 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:44.785 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-02-20T13:13:44.785 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 352 k 2024-02-20T13:13:44.785 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:13:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T13:13:45.100 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-16.2.14-548.g81bd20d6.el8.noarch.rpm 271 kB/s | 85 kB 00:00 2024-02-20T13:13:45.101 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:45.101 INFO:teuthology.orchestra.run.smithi038.stdout:Total 269 kB/s | 85 kB 00:00 2024-02-20T13:13:45.101 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:13:45.109 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:13:45.109 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:13:45.111 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:13:45.114 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:13:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:13:45.199 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T13:13:45.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T13:13:45.671 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T13:13:45.857 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:45.969 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T13:13:46.113 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:46.202 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:13:46.503 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T13:13:46.513 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:46.513 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:46.513 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:13:46.513 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:13:46.513 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:46.513 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:46.739 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T13:13:46.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T13:13:46.961 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T13:13:47.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T13:13:47.373 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T13:13:47.545 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:23 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:47.587 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T13:13:47.844 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T13:13:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T13:13:48.479 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T13:13:48.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 27/32 2024-02-20T13:13:49.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:13:49.217 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:49.219 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:49.219 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-02-20T13:13:49.219 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:49.219 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:13:49.219 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 161 k 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 161 k 2024-02-20T13:13:49.220 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 470 k 2024-02-20T13:13:49.221 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:13:49.614 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-16.2.14-548.g81bd20 408 kB/s | 161 kB 00:00 2024-02-20T13:13:49.614 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:49.615 INFO:teuthology.orchestra.run.smithi038.stdout:Total 406 kB/s | 161 kB 00:00 2024-02-20T13:13:49.615 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:13:49.627 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:13:49.627 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:13:49.679 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:13:49.683 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:13:50.450 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:13:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-20T13:13:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:13:51.260 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-20T13:13:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:13:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:13:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-20T13:13:51.925 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:13:51.942 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-20T13:13:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:13:52.074 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:13:52.404 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:13:52.404 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:52.404 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:13:52.404 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:52.404 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:52.405 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 1/32 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2/32 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 3/32 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 4/32 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 5/32 2024-02-20T13:13:52.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 6/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T13:13:52.476 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T13:13:52.477 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T13:13:52.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:53.373 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T13:13:53.374 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T13:13:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T13:13:53.376 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T13:13:53.376 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:53.376 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:13:53.419 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:29 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:53.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-02-20T13:13:54.438 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:28 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:13:55.032 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:13:55.083 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:55.085 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:13:55.085 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:13:55.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T13:13:55.959 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:13:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:31 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:13:56.007 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:13:56.008 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:13:56.008 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:13:56.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-02-20T13:13:56.864 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:30 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:13:57.643 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 3.1 M 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 24 k 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 15 k 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:57.645 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:13:57.646 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:13:57.646 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-02-20T13:13:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:13:57.646 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.4 M 2024-02-20T13:13:57.646 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 22 M 2024-02-20T13:13:57.646 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:13:57.884 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-grafana-dashboards-16.2.14-548.g81b 167 kB/s | 24 kB 00:00 2024-02-20T13:13:57.901 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): ceph-prometheus-alerts-16.2.14-548.g81bd 91 kB/s | 15 kB 00:00 2024-02-20T13:13:58.026 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2024-02-20T13:13:58.276 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 90 kB/s | 34 kB 00:00 2024-02-20T13:13:58.401 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): ceph-mgr-dashboard-16.2.14-548.g81bd20d6 4.7 MB/s | 3.1 MB 00:00 2024-02-20T13:13:58.412 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 85 k 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:58.413 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:13:58.414 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:13:58.414 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:13:58.414 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:58.414 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-02-20T13:13:58.414 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 352 k 2024-02-20T13:13:58.415 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:13:58.571 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-16.2.14-548.g81bd20d6.el8.noarch.rpm 544 kB/s | 85 kB 00:00 2024-02-20T13:13:58.572 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:58.572 INFO:teuthology.orchestra.run.smithi114.stdout:Total 540 kB/s | 85 kB 00:00 2024-02-20T13:13:58.572 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:13:58.579 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:13:58.579 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:13:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:13:58.584 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:13:58.601 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 341 kB/s | 196 kB 00:00 2024-02-20T13:13:58.605 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:13:58.606 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.6 MB/s | 3.4 MB 00:00 2024-02-20T13:13:58.618 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:13:58.623 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:13:58.629 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:13:58.629 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:13:58.792 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:13:58.796 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:13:59.071 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:13:59.132 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:59.293 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T13:13:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:59.545 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T13:13:59.552 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:59.728 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T13:13:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 4/6 2024-02-20T13:13:59.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-20T13:13:59.933 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:59.933 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:13:59.933 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:13:59.933 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:13:59.933 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:00.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T13:14:00.922 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:00.961 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 5/6 2024-02-20T13:14:01.192 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:14:01.472 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:14:01.472 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 1/6 2024-02-20T13:14:01.472 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 2/6 2024-02-20T13:14:01.472 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 3/6 2024-02-20T13:14:01.472 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T13:14:01.472 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T13:14:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T13:14:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T13:14:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:01.880 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:14:02.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T13:14:02.466 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:02.467 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-02-20T13:14:02.467 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:02.467 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 161 k 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 161 k 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 470 k 2024-02-20T13:14:02.468 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:14:02.907 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:38 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:14:03.246 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-16.2.14-548.g81bd20 207 kB/s | 161 kB 00:00 2024-02-20T13:14:03.246 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:03.247 INFO:teuthology.orchestra.run.smithi114.stdout:Total 207 kB/s | 161 kB 00:00 2024-02-20T13:14:03.247 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:14:03.259 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:14:03.260 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:14:03.311 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:14:03.314 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:14:03.422 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:14:03.607 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:14:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:14:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-20T13:14:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:04.428 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:14:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:14:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:04.428 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:04.561 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 8.6 M 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T13:14:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout:Install 8 Packages 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 37 M 2024-02-20T13:14:04.564 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 210 M 2024-02-20T13:14:04.565 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:14:04.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-02-20T13:14:05.432 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:05.432 INFO:teuthology.orchestra.run.smithi038.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-54 11 MB/s | 8.6 MB 00:00 2024-02-20T13:14:06.926 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:14:06.975 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:06.976 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:14:06.976 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:07.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T13:14:07.828 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:09.383 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:09.384 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:14:09.384 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:09.384 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 3.1 M 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 24 k 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 15 k 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-02-20T13:14:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:09.386 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.4 M 2024-02-20T13:14:09.386 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 22 M 2024-02-20T13:14:09.386 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:14:09.790 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): ceph-prometheus-alerts-16.2.14-548.g81bd 100 kB/s | 15 kB 00:00 2024-02-20T13:14:09.949 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): ceph-grafana-dashboards-16.2.14-548.g81b 79 kB/s | 24 kB 00:00 2024-02-20T13:14:10.007 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 197 kB/s | 43 kB 00:00 2024-02-20T13:14:10.174 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 149 kB/s | 34 kB 00:00 2024-02-20T13:14:10.258 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 800 kB/s | 196 kB 00:00 2024-02-20T13:14:11.008 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): ceph-mgr-dashboard-16.2.14-548.g81bd20d6 2.3 MB/s | 3.1 MB 00:01 2024-02-20T13:14:11.012 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:11.012 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.1 MB/s | 3.4 MB 00:01 2024-02-20T13:14:11.023 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:14:11.034 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:14:11.034 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:14:11.211 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:14:11.214 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:14:11.484 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:14:11.492 INFO:teuthology.orchestra.run.smithi038.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 631 kB/s | 3.7 MB 00:06 2024-02-20T13:14:11.692 INFO:teuthology.orchestra.run.smithi038.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 687 kB/s | 4.7 MB 00:07 2024-02-20T13:14:11.694 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T13:14:11.842 INFO:teuthology.orchestra.run.smithi038.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 643 kB/s | 225 kB 00:00 2024-02-20T13:14:11.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T13:14:12.126 INFO:teuthology.orchestra.run.smithi038.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-02-20T13:14:12.174 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T13:14:12.220 INFO:teuthology.orchestra.run.smithi038.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2024-02-20T13:14:12.337 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 4/6 2024-02-20T13:14:13.376 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 5/6 2024-02-20T13:14:13.820 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:14:14.132 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-20T13:14:14.132 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 1/6 2024-02-20T13:14:14.132 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 2/6 2024-02-20T13:14:14.132 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 3/6 2024-02-20T13:14:14.132 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T13:14:14.132 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T13:14:14.520 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T13:14:14.521 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T13:14:14.521 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:14.521 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:14.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T13:14:14.725 INFO:teuthology.orchestra.run.smithi038.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 493 kB/s | 4.9 MB 00:10 2024-02-20T13:14:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:49 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:16.979 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:16.980 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-02-20T13:14:16.980 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:14:16.980 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-02-20T13:14:16.980 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:14:16.980 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 8.6 M 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-02-20T13:14:16.981 INFO:teuthology.orchestra.run.smithi114.stdout:Install 8 Packages 2024-02-20T13:14:16.982 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:16.982 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 37 M 2024-02-20T13:14:16.982 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 210 M 2024-02-20T13:14:16.982 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:14:18.196 INFO:teuthology.orchestra.run.smithi114.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-54 9.1 MB/s | 8.6 MB 00:00 2024-02-20T13:14:23.423 INFO:teuthology.orchestra.run.smithi114.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 731 kB/s | 3.7 MB 00:05 2024-02-20T13:14:23.773 INFO:teuthology.orchestra.run.smithi114.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 644 kB/s | 225 kB 00:00 2024-02-20T13:14:25.924 INFO:teuthology.orchestra.run.smithi114.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 558 kB/s | 4.7 MB 00:08 2024-02-20T13:14:26.116 INFO:teuthology.orchestra.run.smithi114.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 561 kB/s | 4.9 MB 00:08 2024-02-20T13:14:26.357 INFO:teuthology.orchestra.run.smithi114.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 714 kB/s | 172 kB 00:00 2024-02-20T13:14:26.559 INFO:teuthology.orchestra.run.smithi038.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 976 kB/s | 14 MB 00:14 2024-02-20T13:14:26.560 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:26.560 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.7 MB/s | 37 MB 00:21 2024-02-20T13:14:26.801 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:14:26.820 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:14:26.820 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:14:27.172 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:14:27.175 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:14:27.407 INFO:teuthology.orchestra.run.smithi114.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 435 kB/s | 645 kB 00:01 2024-02-20T13:14:27.746 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:14:28.021 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:14:28.167 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:14:28.325 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:14:28.960 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:14:29.146 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:14:29.649 INFO:teuthology.orchestra.run.smithi114.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 2.4 MB/s | 14 MB 00:05 2024-02-20T13:14:29.650 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:29.650 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.9 MB/s | 37 MB 00:12 2024-02-20T13:14:29.788 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:14:29.883 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:14:29.901 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:14:29.902 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:14:29.958 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:14:30.241 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:14:30.244 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:14:30.679 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:14:30.847 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:14:30.994 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:14:31.130 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T13:14:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:14:31.303 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:14:31.383 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:14:31.898 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T13:14:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:14:32.682 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:14:32.855 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:14:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:14:33.522 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T13:14:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:14:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:14:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T13:14:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:14:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:14:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 1/8 2024-02-20T13:14:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T13:14:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:14:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T13:14:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:14:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T13:14:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T13:14:35.815 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T13:14:35.816 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:35.816 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:14:35.869 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T13:14:35.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-02-20T13:14:36.728 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:12 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:14:37.177 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T13:14:37.402 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:14:37.708 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-20T13:14:37.708 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 1/8 2024-02-20T13:14:37.708 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T13:14:37.709 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T13:14:37.709 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T13:14:37.709 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T13:14:37.709 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T13:14:37.709 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T13:14:38.132 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T13:14:38.133 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T13:14:38.133 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T13:14:38.133 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T13:14:38.133 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T13:14:38.133 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:38.133 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:38.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-02-20T13:14:38.392 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:14:38.394 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:14:38.394 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:38.394 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 27 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T13:14:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-02-20T13:14:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:38.397 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.9 M 2024-02-20T13:14:38.397 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 24 M 2024-02-20T13:14:38.397 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:14:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 175 kB/s | 27 kB 00:00 2024-02-20T13:14:38.858 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 113 kB/s | 20 kB 00:00 2024-02-20T13:14:38.892 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): ceph-mgr-rook-16.2.14-548.g81bd20d6.el8 130 kB/s | 27 kB 00:00 2024-02-20T13:14:38.976 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 887 kB/s | 126 kB 00:00 2024-02-20T13:14:39.001 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 772 kB/s | 110 kB 00:00 2024-02-20T13:14:39.027 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:13 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:39.068 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 881 kB/s | 155 kB 00:00 2024-02-20T13:14:39.112 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 314 kB/s | 43 kB 00:00 2024-02-20T13:14:39.129 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 258 kB/s | 33 kB 00:00 2024-02-20T13:14:39.146 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 210 kB/s | 16 kB 00:00 2024-02-20T13:14:39.196 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-02-20T13:14:39.222 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 744 kB/s | 81 kB 00:00 2024-02-20T13:14:39.238 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.4 MB/s | 61 kB 00:00 2024-02-20T13:14:39.297 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 7.7 MB/s | 1.2 MB 00:00 2024-02-20T13:14:39.297 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:39.297 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2024-02-20T13:14:39.354 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:14:39.370 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:14:39.370 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:14:39.463 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:14:39.466 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:14:40.084 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:14:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T13:14:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T13:14:40.576 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 27 k 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T13:14:40.578 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-02-20T13:14:40.579 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:40.580 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 1.9 M 2024-02-20T13:14:40.580 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 24 M 2024-02-20T13:14:40.580 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:14:40.782 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T13:14:41.005 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): ceph-mgr-rook-16.2.14-548.g81bd20d6.el8 195 kB/s | 27 kB 00:00 2024-02-20T13:14:41.044 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T13:14:41.055 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 139 kB/s | 27 kB 00:00 2024-02-20T13:14:41.080 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 92 kB/s | 20 kB 00:00 2024-02-20T13:14:41.147 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 889 kB/s | 126 kB 00:00 2024-02-20T13:14:41.222 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 776 kB/s | 110 kB 00:00 2024-02-20T13:14:41.245 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T13:14:41.248 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 927 kB/s | 155 kB 00:00 2024-02-20T13:14:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 364 kB/s | 43 kB 00:00 2024-02-20T13:14:41.465 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 136 kB/s | 33 kB 00:00 2024-02-20T13:14:41.471 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T13:14:41.490 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 67 kB/s | 16 kB 00:00 2024-02-20T13:14:41.599 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 243 kB/s | 81 kB 00:00 2024-02-20T13:14:41.666 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 366 kB/s | 64 kB 00:00 2024-02-20T13:14:41.691 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 668 kB/s | 61 kB 00:00 2024-02-20T13:14:41.758 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T13:14:41.883 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.8 MB/s | 1.2 MB 00:00 2024-02-20T13:14:41.883 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:41.884 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2024-02-20T13:14:41.940 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:14:41.956 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:14:41.956 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:14:42.036 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T13:14:42.044 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:14:42.047 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:14:42.648 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:14:42.887 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T13:14:43.119 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T13:14:43.129 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T13:14:43.361 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T13:14:43.435 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T13:14:43.608 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T13:14:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T13:14:43.831 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T13:14:43.953 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T13:14:44.079 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T13:14:44.119 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:14:44.355 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T13:14:44.462 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 1/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T13:14:44.463 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T13:14:44.580 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T13:14:45.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:45.083 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:14:45.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T13:14:45.598 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T13:14:46.017 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:22 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:14:46.207 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T13:14:46.419 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T13:14:46.616 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T13:14:46.775 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:14:47.090 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-20T13:14:47.090 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 1/13 2024-02-20T13:14:47.090 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T13:14:47.090 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T13:14:47.090 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T13:14:47.091 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T13:14:47.564 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T13:14:47.564 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:47.564 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:14:47.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:47.565 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:47.681 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 116 k 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:14:47.682 INFO:teuthology.orchestra.run.smithi038.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-02-20T13:14:47.683 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:47.684 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 5.9 M 2024-02-20T13:14:47.684 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 24 M 2024-02-20T13:14:47.684 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:14:47.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T13:14:48.321 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-16.2.14-548.g81bd20d6.e 612 kB/s | 116 kB 00:00 2024-02-20T13:14:48.455 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2024-02-20T13:14:48.466 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:22 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:48.505 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.5 MB/s | 578 kB 00:00 2024-02-20T13:14:48.572 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.0 MB/s | 355 kB 00:00 2024-02-20T13:14:48.664 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 308 kB/s | 49 kB 00:00 2024-02-20T13:14:49.373 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 3.9 MB/s | 4.8 MB 00:01 2024-02-20T13:14:49.375 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:49.375 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.5 MB/s | 5.9 MB 00:01 2024-02-20T13:14:49.430 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:14:49.441 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:14:49.441 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:14:49.552 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:14:49.556 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:14:49.836 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:14:50.011 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 116 k 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:50.012 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:14:50.013 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:50.013 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-02-20T13:14:50.013 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:50.013 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 5.9 M 2024-02-20T13:14:50.013 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 24 M 2024-02-20T13:14:50.013 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:14:50.082 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T13:14:50.259 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): ceph-mgr-cephadm-16.2.14-548.g81bd20d6.e 637 kB/s | 116 kB 00:00 2024-02-20T13:14:50.382 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T13:14:50.426 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.6 MB/s | 578 kB 00:00 2024-02-20T13:14:50.452 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 110 kB/s | 21 kB 00:00 2024-02-20T13:14:50.611 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2024-02-20T13:14:50.636 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 266 kB/s | 49 kB 00:00 2024-02-20T13:14:50.719 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.5 MB/s | 4.8 MB 00:00 2024-02-20T13:14:50.720 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:50.720 INFO:teuthology.orchestra.run.smithi114.stdout:Total 8.3 MB/s | 5.9 MB 00:00 2024-02-20T13:14:50.770 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:14:50.780 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:14:50.780 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:14:50.879 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:14:50.978 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:14:51.178 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:14:51.368 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T13:14:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T13:14:51.717 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T13:14:51.743 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T13:14:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T13:14:52.235 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:14:52.537 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:14:52.537 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/6 2024-02-20T13:14:52.537 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T13:14:52.537 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T13:14:52.537 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T13:14:52.538 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T13:14:52.659 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T13:14:52.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-20T13:14:52.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-20T13:14:52.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-20T13:14:52.939 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:52.939 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:14:53.012 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T13:14:53.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-02-20T13:14:53.250 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T13:14:53.466 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:14:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-20T13:14:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/6 2024-02-20T13:14:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T13:14:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T13:14:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T13:14:53.753 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T13:14:53.858 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:29 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:14:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T13:14:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:54.163 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:14:54.163 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:54.164 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:54.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-02-20T13:14:55.085 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:29 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:14:55.515 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 911 k 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:14:55.516 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-02-20T13:14:55.517 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:55.517 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 994 k 2024-02-20T13:14:55.517 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.8 M 2024-02-20T13:14:55.517 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:14:55.973 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 370 kB/s | 83 kB 00:00 2024-02-20T13:14:56.407 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-16.2.14-548.g81bd20d6.el8.x86_ 1.4 MB/s | 911 kB 00:00 2024-02-20T13:14:56.407 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:56.408 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.1 MB/s | 994 kB 00:00 2024-02-20T13:14:56.420 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:14:56.444 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:14:56.444 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:14:56.544 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:14:56.547 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:14:56.637 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 911 k 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-02-20T13:14:56.638 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:56.639 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 994 k 2024-02-20T13:14:56.639 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.8 M 2024-02-20T13:14:56.639 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:14:56.725 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:14:56.987 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 534 kB/s | 83 kB 00:00 2024-02-20T13:14:57.029 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T13:14:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:14:57.287 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-16.2.14-548.g81bd20d6.el8.x86_ 2.0 MB/s | 911 kB 00:00 2024-02-20T13:14:57.288 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:14:57.288 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.5 MB/s | 994 kB 00:00 2024-02-20T13:14:57.298 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:14:57.319 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:14:57.319 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:14:57.408 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:14:57.411 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:14:57.581 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:14:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:14:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/2 2024-02-20T13:14:57.864 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T13:14:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T13:14:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:58.104 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:14:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T13:14:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:14:58.104 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:14:58.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:14:58.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-02-20T13:14:58.495 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-20T13:14:58.495 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/2 2024-02-20T13:14:58.877 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T13:14:58.877 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:58.878 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:14:58.878 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T13:14:58.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:14:58.878 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:14:59.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-02-20T13:14:59.112 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:35 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:14:59.852 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:33 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:00.785 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:00.785 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:00.785 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-02-20T13:15:00.785 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 131 k 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:00.786 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 131 k 2024-02-20T13:15:00.787 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 432 k 2024-02-20T13:15:00.787 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:15:01.088 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-16.2.14-548.g81bd20d6.el8.x86_64 436 kB/s | 131 kB 00:00 2024-02-20T13:15:01.088 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:01.089 INFO:teuthology.orchestra.run.smithi038.stdout:Total 435 kB/s | 131 kB 00:00 2024-02-20T13:15:01.089 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:15:01.094 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:15:01.094 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:15:01.149 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:15:01.152 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:15:01.264 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 131 k 2024-02-20T13:15:01.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:01.395 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:15:01.395 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:01.395 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:15:01.395 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:01.395 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 131 k 2024-02-20T13:15:01.396 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 432 k 2024-02-20T13:15:01.396 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:15:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:01.874 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-16.2.14-548.g81bd20d6.el8.x86_64 274 kB/s | 131 kB 00:00 2024-02-20T13:15:01.875 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:01.875 INFO:teuthology.orchestra.run.smithi114.stdout:Total 273 kB/s | 131 kB 00:00 2024-02-20T13:15:01.875 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:15:01.881 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:15:01.881 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:15:01.937 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:15:01.940 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:15:02.061 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:15:02.145 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:02.146 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:02.146 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:15:02.146 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:02.146 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:02.146 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:02.289 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:02.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-02-20T13:15:02.567 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:02.946 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:02.946 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:02.946 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:15:02.946 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:02.946 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:02.947 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:03.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-02-20T13:15:03.171 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:39 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:03.938 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:37 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:04.781 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:04.832 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:04.834 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:04.834 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:04.930 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-02-20T13:15:05.443 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:05.489 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:05.490 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:05.491 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:05.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-02-20T13:15:05.716 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:41 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:06.348 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:40 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:07.411 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 25 k 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:15:07.412 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:07.413 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 25 k 2024-02-20T13:15:07.413 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 107 k 2024-02-20T13:15:07.413 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:15:07.560 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-16.2.14-548.g81bd20d6.el8.x86_6 174 kB/s | 25 kB 00:00 2024-02-20T13:15:07.561 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:07.561 INFO:teuthology.orchestra.run.smithi038.stdout:Total 173 kB/s | 25 kB 00:00 2024-02-20T13:15:07.561 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:15:07.564 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:15:07.565 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:15:07.567 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:15:07.570 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:15:07.619 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:15:07.815 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:07.917 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:07.917 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:07.917 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 25 k 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 25 k 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 107 k 2024-02-20T13:15:07.918 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:15:08.076 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:08.464 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:08.464 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:08.464 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:15:08.464 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:08.464 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:08.464 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:08.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-02-20T13:15:08.738 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-16.2.14-548.g81bd20d6.el8.x86_6 31 kB/s | 25 kB 00:00 2024-02-20T13:15:08.739 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:08.739 INFO:teuthology.orchestra.run.smithi114.stdout:Total 31 kB/s | 25 kB 00:00 2024-02-20T13:15:08.740 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:15:08.744 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:15:08.744 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:15:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:15:08.750 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:15:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:15:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:09.373 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:45 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:09.676 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:15:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:09.676 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:09.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-02-20T13:15:10.556 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:44 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:10.993 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:11.044 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:11.046 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:11.046 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:11.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-02-20T13:15:11.933 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:47 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:12.067 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:12.114 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:12.115 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:12.115 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:12.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-02-20T13:15:12.976 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:46 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:13.555 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:13.605 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:13.607 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:13.618 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:13.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-02-20T13:15:14.470 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:14.515 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:50 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:14.517 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:14.518 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:14.518 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:14.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-02-20T13:15:15.371 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:49 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:16.200 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:16.249 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:16.251 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:16.251 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:16.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-02-20T13:15:16.889 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:16.936 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:16.937 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:16.937 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:17.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-02-20T13:15:17.142 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:53 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:17.782 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:51 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:18.759 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:18.810 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:18.811 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:18.811 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:18.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-02-20T13:15:19.286 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:19.333 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:19.335 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:19.335 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:19.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-02-20T13:15:19.693 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:55 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:20.246 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:54 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:21.317 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:21.367 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:21.368 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:21.369 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:21.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-02-20T13:15:21.745 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:21.791 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:21.792 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:21.792 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:21.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-02-20T13:15:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:58 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:22.638 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:56 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:23.884 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:23.935 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:23.936 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-02-20T13:15:23.936 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:24.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-02-20T13:15:24.144 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-20T13:15:24.191 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:24.192 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-20T13:15:24.192 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:24.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-02-20T13:15:24.824 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:00 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:25.053 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:59 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:26.494 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:26.494 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 88 k 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:15:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:26.496 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 88 k 2024-02-20T13:15:26.496 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 240 k 2024-02-20T13:15:26.496 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:15:26.615 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 88 k 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:15:26.616 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:26.617 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:15:26.617 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:26.617 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 88 k 2024-02-20T13:15:26.617 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 240 k 2024-02-20T13:15:26.617 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:15:26.850 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-16.2.14-548.g81bd20d6.el8.x86_64.rpm 250 kB/s | 88 kB 00:00 2024-02-20T13:15:26.851 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:26.851 INFO:teuthology.orchestra.run.smithi038.stdout:Total 249 kB/s | 88 kB 00:00 2024-02-20T13:15:26.851 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:15:26.862 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:15:26.862 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:15:26.916 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:15:26.919 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:15:26.996 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-16.2.14-548.g81bd20d6.el8.x86_64.rpm 234 kB/s | 88 kB 00:00 2024-02-20T13:15:26.996 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:26.996 INFO:teuthology.orchestra.run.smithi114.stdout:Total 233 kB/s | 88 kB 00:00 2024-02-20T13:15:26.997 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:15:27.006 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:15:27.006 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:15:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:15:27.055 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:15:27.058 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:15:27.166 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:15:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:27.344 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:27.528 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:27.618 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:27.932 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:27.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:27.932 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:15:27.932 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:27.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:27.933 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:27.999 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:27.999 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:27.999 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:15:27.999 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:27.999 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:27.999 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:28.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-02-20T13:15:28.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-02-20T13:15:28.949 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:04 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:29.001 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:03:03 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:30.554 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:30.554 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:15:30.555 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:30.556 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.7 M 2024-02-20T13:15:30.556 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 15 M 2024-02-20T13:15:30.556 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:15:30.612 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:15:30.613 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:30.614 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.7 M 2024-02-20T13:15:30.614 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 15 M 2024-02-20T13:15:30.614 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:15:31.340 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-16.2.14-548.g81bd20d6.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-02-20T13:15:31.340 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:31.341 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2024-02-20T13:15:31.341 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:15:31.353 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:15:31.353 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:15:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:15:31.420 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:15:31.529 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-16.2.14-548.g81bd20d6.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2024-02-20T13:15:31.530 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:31.530 INFO:teuthology.orchestra.run.smithi038.stdout:Total 4.0 MB/s | 3.7 MB 00:00 2024-02-20T13:15:31.530 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:15:31.543 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:15:31.543 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:15:31.613 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:15:31.616 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:15:31.855 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:15:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:15:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:32.334 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:32.495 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:32.762 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:32.881 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:32.881 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:32.881 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:15:32.881 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:32.881 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:32.881 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:33.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-02-20T13:15:33.187 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:33.187 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:33.187 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:15:33.187 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:33.187 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:33.187 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:33.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-02-20T13:15:33.879 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:03:07 ago on Tue 20 Feb 2024 01:12:26 PM UTC. 2024-02-20T13:15:34.238 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:10 ago on Tue 20 Feb 2024 01:12:24 PM UTC. 2024-02-20T13:15:35.433 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-20T13:15:35.434 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:35.434 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-20T13:15:35.434 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:35.434 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 175 k 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 175 k 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 523 k 2024-02-20T13:15:35.435 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-20T13:15:35.758 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-16.2.14-548.g81bd20d6.el8.x86_64.rpm 543 kB/s | 175 kB 00:00 2024-02-20T13:15:35.759 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:35.759 INFO:teuthology.orchestra.run.smithi114.stdout:Total 540 kB/s | 175 kB 00:00 2024-02-20T13:15:35.759 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-20T13:15:35.770 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-20T13:15:35.770 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-20T13:15:35.819 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-20T13:15:35.822 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-20T13:15:35.913 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-20T13:15:35.913 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:35.913 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-02-20T13:15:35.913 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 175 k 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-02-20T13:15:35.914 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:35.915 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 175 k 2024-02-20T13:15:35.915 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 523 k 2024-02-20T13:15:35.915 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-20T13:15:35.940 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-20T13:15:36.127 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-16.2.14-548.g81bd20d6.el8.x86_64.rpm 825 kB/s | 175 kB 00:00 2024-02-20T13:15:36.128 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-20T13:15:36.128 INFO:teuthology.orchestra.run.smithi038.stdout:Total 820 kB/s | 175 kB 00:00 2024-02-20T13:15:36.128 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-20T13:15:36.139 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-20T13:15:36.140 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-20T13:15:36.142 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:36.193 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-20T13:15:36.196 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-20T13:15:36.313 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-20T13:15:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:36.512 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:36.805 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:36.825 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:36.826 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:36.826 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-20T13:15:36.826 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:36.826 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:15:36.826 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-20T13:15:37.085 DEBUG:teuthology.parallel:result is None 2024-02-20T13:15:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-20T13:15:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-20T13:15:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-20T13:15:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:15:37.217 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-20T13:15:37.436 DEBUG:teuthology.parallel:result is None 2024-02-20T13:15:37.437 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:15:37.437 INFO:teuthology.packaging:ref: None 2024-02-20T13:15:37.437 INFO:teuthology.packaging:tag: None 2024-02-20T13:15:37.437 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:15:37.437 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:15:37.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:15:37.777 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T13:15:37.801 INFO:teuthology.orchestra.run.smithi038.stdout:16.2.14-548.g81bd20d6.el8 2024-02-20T13:15:37.802 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-548.g81bd20d6.el8 2024-02-20T13:15:37.802 INFO:teuthology.task.install:The correct ceph version 16.2.14-548.g81bd20d6 is installed. 2024-02-20T13:15:37.804 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T13:15:37.804 INFO:teuthology.packaging:ref: None 2024-02-20T13:15:37.804 INFO:teuthology.packaging:tag: None 2024-02-20T13:15:37.804 INFO:teuthology.packaging:branch: pacific 2024-02-20T13:15:37.804 INFO:teuthology.packaging:sha1: b0bd01b8442d66b8a31f8cb3050334b026de2f3b 2024-02-20T13:15:37.805 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-02-20T13:15:38.052 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T13:15:38.076 INFO:teuthology.orchestra.run.smithi114.stdout:16.2.14-548.g81bd20d6.el8 2024-02-20T13:15:38.076 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-548.g81bd20d6.el8 2024-02-20T13:15:38.076 INFO:teuthology.task.install:The correct ceph version 16.2.14-548.g81bd20d6 is installed. 2024-02-20T13:15:38.078 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-20T13:15:38.078 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:15:38.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T13:15:38.109 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:15:38.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T13:15:38.144 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-20T13:15:38.144 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:15:38.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T13:15:38.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T13:15:38.247 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:15:38.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T13:15:38.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T13:15:38.342 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-20T13:15:38.342 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:15:38.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T13:15:38.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T13:15:38.441 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:15:38.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T13:15:38.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T13:15:38.535 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-20T13:15:38.535 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:15:38.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T13:15:38.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T13:15:38.633 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:15:38.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T13:15:38.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T13:15:38.727 INFO:teuthology.run_tasks:Running task print... 2024-02-20T13:15:38.740 INFO:teuthology.task.print:**** done install task... 2024-02-20T13:15:38.740 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-20T13:15:38.866 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b0bd01b8442d66b8a31f8cb3050334b026de2f3b'} 2024-02-20T13:15:38.866 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2024-02-20T13:15:38.867 INFO:tasks.cephadm:Cluster fsid is 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:15:38.867 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-20T13:15:38.867 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-20T13:15:38.867 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi038': '172.21.15.38', 'mon.smithi114': '172.21.15.114'} 2024-02-20T13:15:38.867 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-20T13:15:38.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-02-20T13:15:38.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-02-20T13:15:38.926 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2024-02-20T13:15:38.926 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T13:15:39.212 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 20 13:15 /home/ubuntu/cephtest/cephadm 2024-02-20T13:15:39.213 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T13:15:39.402 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 20 13:15 /home/ubuntu/cephtest/cephadm 2024-02-20T13:15:39.403 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T13:15:39.427 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T13:15:39.476 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2024-02-20T13:15:39.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2024-02-20T13:15:39.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2024-02-20T13:15:40.726 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:15:40.855 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:16:05.246 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-02-20T13:16:05.247 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2024-02-20T13:16:05.247 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2024-02-20T13:16:05.247 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-02-20T13:16:05.247 INFO:teuthology.orchestra.run.smithi038.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2024-02-20T13:16:05.247 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-02-20T13:16:05.247 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-02-20T13:16:06.072 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-20T13:16:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2024-02-20T13:16:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2024-02-20T13:16:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-02-20T13:16:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2024-02-20T13:16:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-02-20T13:16:06.073 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-20T13:16:06.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-02-20T13:16:06.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-02-20T13:16:06.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-02-20T13:16:06.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-02-20T13:16:06.231 INFO:tasks.cephadm:Writing seed config... 2024-02-20T13:16:06.232 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-20T13:16:06.232 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-20T13:16:06.232 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-20T13:16:06.232 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-20T13:16:06.232 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-20T13:16:06.232 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-20T13:16:06.233 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-20T13:16:06.234 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-20T13:16:06.235 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-20T13:16:06.235 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-20T13:16:06.235 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-20T13:16:06.235 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-20T13:16:06.235 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:16:06.236 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-20T13:16:06.265 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 24090638-cff2-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-20T13:16:06.266 DEBUG:teuthology.orchestra.run.smithi038:mon.smithi038> sudo journalctl -f -n 0 -u ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038.service 2024-02-20T13:16:06.309 INFO:tasks.cephadm:Bootstrapping... 2024-02-20T13:16:06.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid 24090638-cff2-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-20T13:16:06.340 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:-- Logs begin at Tue 2024-02-20 13:04:48 UTC. -- 2024-02-20T13:16:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:-------------------------------------------------------------------------------- 2024-02-20T13:16:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', '24090638-cff2-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2024-02-20T13:16:06.553 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 4.9.3-dev 2024-02-20T13:16:06.565 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying podman|docker is present... 2024-02-20T13:16:06.586 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 4.9.3-dev 2024-02-20T13:16:06.596 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying lvm2 is present... 2024-02-20T13:16:06.597 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying time synchronization is in place... 2024-02-20T13:16:06.603 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T13:16:06.609 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2024-02-20T13:16:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: enabled 2024-02-20T13:16:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: active 2024-02-20T13:16:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:Unit chronyd.service is enabled and running 2024-02-20T13:16:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:Repeating the final host check... 2024-02-20T13:16:06.642 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 4.9.3-dev 2024-02-20T13:16:06.650 INFO:teuthology.orchestra.run.smithi038.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-20T13:16:06.650 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl is present 2024-02-20T13:16:06.650 INFO:teuthology.orchestra.run.smithi038.stderr:lvcreate is present 2024-02-20T13:16:06.655 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T13:16:06.664 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2024-02-20T13:16:06.667 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: enabled 2024-02-20T13:16:06.672 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: active 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Unit chronyd.service is enabled and running 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Host looks OK 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Cluster fsid: 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Acquiring lock 139670000663016 on /run/cephadm/24090638-cff2-11ee-95bb-87774f69a715.lock 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Lock 139670000663016 acquired on /run/cephadm/24090638-cff2-11ee-95bb-87774f69a715.lock 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying IP 172.21.15.38 port 3300 ... 2024-02-20T13:16:06.674 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying IP 172.21.15.38 port 6789 ... 2024-02-20T13:16:06.675 INFO:teuthology.orchestra.run.smithi038.stderr:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-02-20T13:16:06.679 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-02-20T13:16:06.679 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-02-20T13:16:06.681 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-20T13:16:06.681 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-02-20T13:16:06.681 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-02-20T13:16:06.684 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-20T13:16:06.684 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-20T13:16:06.684 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T13:16:06.684 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-20T13:16:06.684 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15fc/64 scope link 2024-02-20T13:16:06.684 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T13:16:06.685 INFO:teuthology.orchestra.run.smithi038.stderr:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-02-20T13:16:06.685 INFO:teuthology.orchestra.run.smithi038.stderr:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-02-20T13:16:06.685 INFO:teuthology.orchestra.run.smithi038.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-20T13:16:06.686 INFO:teuthology.orchestra.run.smithi038.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-20T13:16:06.686 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:16:06.766 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2024-02-20T13:16:13.266 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Getting image source signatures 2024-02-20T13:16:13.266 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-20T13:16:13.266 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Copying blob sha256:a997af56d24c50f31dbc8bd557a1c2adeb9ea675d7d7142f1a37cb629070c246 2024-02-20T13:16:13.266 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Copying config sha256:46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2024-02-20T13:16:13.479 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Writing manifest to image destination 2024-02-20T13:16:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2024-02-20T13:16:14.161 INFO:teuthology.orchestra.run.smithi038.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2024-02-20T13:16:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2024-02-20T13:16:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:Extracting ceph user uid/gid from container image... 2024-02-20T13:16:15.382 INFO:teuthology.orchestra.run.smithi038.stderr:stat: 167 167 2024-02-20T13:16:16.008 INFO:teuthology.orchestra.run.smithi038.stderr:Creating initial keys... 2024-02-20T13:16:16.513 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-authtool: AQAgptRlcldcGRAAwcvRoGkuxVMSopRdTOLkWw== 2024-02-20T13:16:17.594 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-authtool: AQAhptRlxL7DHhAAblcOxhOTjBdBypfBBKkkIQ== 2024-02-20T13:16:18.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-authtool: AQAiptRlQ/PXJhAAneN4x6UoxpDzWWvAhKulBw== 2024-02-20T13:16:19.332 INFO:teuthology.orchestra.run.smithi038.stderr:Creating initial monmap... 2024-02-20T13:16:19.883 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T13:16:19.883 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:19.883 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T13:16:20.385 INFO:teuthology.orchestra.run.smithi038.stderr:monmaptool for smithi038 [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T13:16:20.386 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: set fsid to 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:20.386 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T13:16:20.386 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:16:20.386 INFO:teuthology.orchestra.run.smithi038.stderr:Creating mon... 2024-02-20T13:16:21.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.882+0000 7f1720c04700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.882+0000 7f1720c04700 1 imported monmap: 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: fsid 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: last_changed 2024-02-20T13:16:19.788530+0000 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: created 2024-02-20T13:16:19.788530+0000 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.smithi038 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.882+0000 7f1720c04700 0 /usr/bin/ceph-mon: set fsid to 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:21.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Compile date Jul 8 2021 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: DB SUMMARY 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi038/store.db dir, Total Num: 0, files: 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi038/store.db: 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.create_if_missing: 1 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.env: 0x56277df9d1c0 2024-02-20T13:16:21.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.fs: Posix File System 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.info_log: 0x56277ec16c20 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.statistics: (nil) 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.use_fsync: 0 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T13:16:21.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.db_log_dir: 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi038/store.db 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T13:16:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.write_buffer_manager: 0x56277ec1f530 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.unordered_write: 0 2024-02-20T13:16:21.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.row_cache: None 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.wal_filter: None 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T13:16:21.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T13:16:21.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.max_open_files: -1 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Compression algorithms supported: 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kZSTD supported: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T13:16:21.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.883+0000 7f1720c04700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.884+0000 7f1720c04700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000001 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.merge_operator: 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_filter: None 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T13:16:21.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56277eb30160) 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache: 0x56277eb66d10 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-20T13:16:21.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T13:16:21.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression: NoCompression 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.num_levels: 7 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T13:16:21.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T13:16:21.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T13:16:21.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T13:16:21.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T13:16:21.174 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.ttl: 2592000 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.886+0000 7f1720c04700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.887+0000 7f1720c04700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.887+0000 7f1720c04700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.888+0000 7f1720c04700 4 rocksdb: DB pointer 0x56277ec2b800 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.888+0000 7f1709a8d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.888+0000 7f1709a8d700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T13:16:21.175 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:16:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:16:21.177 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:16:21.178 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2024-02-20T13:16:21.179 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.889+0000 7f1720c04700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-20T13:16:21.179 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.889+0000 7f1720c04700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-20T13:16:21.179 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2024-02-20T13:16:20.889+0000 7f1720c04700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi038 for mon.smithi038 2024-02-20T13:16:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:create mon.smithi038 on 2024-02-20T13:16:22.226 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-20T13:16:22.329 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-20T13:16:22.430 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-24090638-cff2-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-24090638-cff2-11ee-95bb-87774f69a715.target. 2024-02-20T13:16:22.430 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-24090638-cff2-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-24090638-cff2-11ee-95bb-87774f69a715.target. 2024-02-20T13:16:22.647 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to reset failed state of unit ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038.service: Unit ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038.service not loaded. 2024-02-20T13:16:22.653 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/ceph-24090638-cff2-11ee-95bb-87774f69a715.target.wants/ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038.service → /etc/systemd/system/ceph-24090638-cff2-11ee-95bb-87774f69a715@.service. 2024-02-20T13:16:22.782 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:22 smithi038 systemd[1]: Starting Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715... 2024-02-20T13:16:23.477 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2024-02-20T13:16:23.485 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2024-02-20T13:16:23.486 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2024-02-20T13:16:23.486 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T13:16:23.486 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mon to start... 2024-02-20T13:16:23.486 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mon... 2024-02-20T13:16:23.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.414289+0000 mon.smithi038 (mon.0) 0 : [INF] mkfs 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.396217+0000 mon.smithi038 (mon.0) 1 : cluster [INF] mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2024-02-20T13:16:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:23 smithi038 bash[37856]: 649d1c82a1e08e653741a7c638a025bbb22ae06e27458357893aa89d4516cfeb 2024-02-20T13:16:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:23 smithi038 systemd[1]: Started Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715. 2024-02-20T13:16:23.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.981+0000 7fdbe5b42700 1 Processor -- start 2024-02-20T13:16:23.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.981+0000 7fdbe5b42700 1 -- start start 2024-02-20T13:16:23.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.982+0000 7fdbe5b42700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 0x7fdbe00febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:23.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.982+0000 7fdbe5b42700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00ff0d0 con 0x7fdbe0027670 2024-02-20T13:16:23.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.982+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 0x7fdbe00febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:23.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.982+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 0x7fdbe00febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53502/0 (socket says 172.21.15.38:53502) 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.982+0000 7fdbdf7fe700 1 -- 172.21.15.38:0/3113960203 learned_addr learned my addr 172.21.15.38:0/3113960203 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.982+0000 7fdbdf7fe700 1 -- 172.21.15.38:0/3113960203 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00ff210 con 0x7fdbe0027670 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbdf7fe700 1 --2- 172.21.15.38:0/3113960203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 0x7fdbe00febe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdbd000b5f0 tx=0x7fdbd0006ee0).ready entity=mon.0 client_cookie=ed7a21412455a584 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbde7fc700 1 -- 172.21.15.38:0/3113960203 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbd000d600 con 0x7fdbe0027670 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbde7fc700 1 -- 172.21.15.38:0/3113960203 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd000d760 con 0x7fdbe0027670 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbde7fc700 1 -- 172.21.15.38:0/3113960203 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbd000da30 con 0x7fdbe0027670 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 -- 172.21.15.38:0/3113960203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 msgr2=0x7fdbe00febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 --2- 172.21.15.38:0/3113960203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 0x7fdbe00febe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdbd000b5f0 tx=0x7fdbd0006ee0).stop 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 -- 172.21.15.38:0/3113960203 shutdown_connections 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 --2- 172.21.15.38:0/3113960203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe0027670 0x7fdbe00febe0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 -- 172.21.15.38:0/3113960203 >> 172.21.15.38:0/3113960203 conn(0x7fdbe00f6680 msgr2=0x7fdbe00f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 -- 172.21.15.38:0/3113960203 shutdown_connections 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.983+0000 7fdbe5b42700 1 -- 172.21.15.38:0/3113960203 wait complete. 2024-02-20T13:16:23.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbe5b42700 1 Processor -- start 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbe5b42700 1 -- start start 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbe5b42700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 0x7fdbe010aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbe5b42700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe010af50 con 0x7fdbe010a640 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 0x7fdbe010aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 0x7fdbe010aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53518/0 (socket says 172.21.15.38:53518) 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.984+0000 7fdbdf7fe700 1 -- 172.21.15.38:0/383685611 learned_addr learned my addr 172.21.15.38:0/383685611 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbdf7fe700 1 -- 172.21.15.38:0/383685611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd000b200 con 0x7fdbe010a640 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbdf7fe700 1 --2- 172.21.15.38:0/383685611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 0x7fdbe010aa60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdbd0006f20 tx=0x7fdbd0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbd0011b00 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0011c80 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe010b150 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbd001d430 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe010bca0 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fdbd000f070 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.985+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdbd001dbb0 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.986+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe00591e0 con 0x7fdbe010a640 2024-02-20T13:16:23.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:23.988+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdbd0016070 con 0x7fdbe010a640 2024-02-20T13:16:24.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.058+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fdbe0045c30 con 0x7fdbe010a640 2024-02-20T13:16:24.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.058+0000 7fdbc7fff700 1 -- 172.21.15.38:0/383685611 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fdbe0045c30 con 0x7fdbe010a640 2024-02-20T13:16:24.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: cluster: 2024-02-20T13:16:24.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: id: 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:24.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-20T13:16:24.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: services: 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi038 (age 0.646217s) 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: data: 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: pgs: 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 msgr2=0x7fdbe010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 --2- 172.21.15.38:0/383685611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 0x7fdbe010aa60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdbd0006f20 tx=0x7fdbd0016000).stop 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 shutdown_connections 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 --2- 172.21.15.38:0/383685611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbe010a640 0x7fdbe010aa60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:24.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 >> 172.21.15.38:0/383685611 conn(0x7fdbe00f6680 msgr2=0x7fdbe00f8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:24.064 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 shutdown_connections 2024-02-20T13:16:24.064 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:24.059+0000 7fdbe5b42700 1 -- 172.21.15.38:0/383685611 wait complete. 2024-02-20T13:16:24.672 INFO:teuthology.orchestra.run.smithi038.stderr:mon is available 2024-02-20T13:16:24.672 INFO:teuthology.orchestra.run.smithi038.stderr:Assimilating anything we can from ceph.conf... 2024-02-20T13:16:24.675 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.413672+0000 mon.smithi038 (mon.0) 2 : cluster [INF] mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2024-02-20T13:16:24.675 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.413944+0000 mon.smithi038 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} 2024-02-20T13:16:24.675 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.415646+0000 mon.smithi038 (mon.0) 4 : cluster [DBG] fsmap 2024-02-20T13:16:24.675 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.422478+0000 mon.smithi038 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T13:16:24.675 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: cluster 2024-02-20T13:16:23.422954+0000 mon.smithi038 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T13:16:24.675 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: audit 2024-02-20T13:16:24.059859+0000 mon.smithi038 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.38:0/383685611' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.259+0000 7f88537ec700 1 Processor -- start 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.259+0000 7f88537ec700 1 -- start start 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f88537ec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c0fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f88537ec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c0fb7e0 con 0x7f884c0faed0 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f8851588700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c0fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f8851588700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c0fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53528/0 (socket says 172.21.15.38:53528) 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f8851588700 1 -- 172.21.15.38:0/3135343011 learned_addr learned my addr 172.21.15.38:0/3135343011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f8851588700 1 -- 172.21.15.38:0/3135343011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c0fb920 con 0x7f884c0faed0 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.260+0000 7f8851588700 1 --2- 172.21.15.38:0/3135343011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c0fb2f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f884000b870 tx=0x7f8840006f70).ready entity=mon.0 client_cookie=632605db9dfeb1e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f883ffff700 1 -- 172.21.15.38:0/3135343011 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884000da20 con 0x7f884c0faed0 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f883ffff700 1 -- 172.21.15.38:0/3135343011 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884000db80 con 0x7f884c0faed0 2024-02-20T13:16:25.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f883ffff700 1 -- 172.21.15.38:0/3135343011 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884000de50 con 0x7f884c0faed0 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 -- 172.21.15.38:0/3135343011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 msgr2=0x7f884c0fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 --2- 172.21.15.38:0/3135343011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c0fb2f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f884000b870 tx=0x7f8840006f70).stop 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 -- 172.21.15.38:0/3135343011 shutdown_connections 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 --2- 172.21.15.38:0/3135343011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c0fb2f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 -- 172.21.15.38:0/3135343011 >> 172.21.15.38:0/3135343011 conn(0x7f884c0f6680 msgr2=0x7f884c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 -- 172.21.15.38:0/3135343011 shutdown_connections 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.261+0000 7f88537ec700 1 -- 172.21.15.38:0/3135343011 wait complete. 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f88537ec700 1 Processor -- start 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f88537ec700 1 -- start start 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f88537ec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f88537ec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c108830 con 0x7f884c0faed0 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f8851588700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f8851588700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53542/0 (socket says 172.21.15.38:53542) 2024-02-20T13:16:25.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.262+0000 7f8851588700 1 -- 172.21.15.38:0/1528408738 learned_addr learned my addr 172.21.15.38:0/1528408738 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f8851588700 1 -- 172.21.15.38:0/1528408738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884000b480 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f8851588700 1 --2- 172.21.15.38:0/1528408738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c108340 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8840006c70 tx=0x7f8840006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884001b110 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884001b270 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884c108a30 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88400105d0 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884c108ec0 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f884002c070 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.263+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8840019450 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.264+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884c045bb0 con 0x7f884c0faed0 2024-02-20T13:16:25.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.266+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f88400449c0 con 0x7f884c0faed0 2024-02-20T13:16:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.334+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f884c06b250 con 0x7f884c0faed0 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.338+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 7 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8840010cc0 con 0x7f884c0faed0 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.338+0000 7f883dffb700 1 -- 172.21.15.38:0/1528408738 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7f8840033030 con 0x7f884c0faed0 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [global] 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: fsid = 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-20T13:16:25.342 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [mgr] 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [osd] 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-20T13:16:25.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 msgr2=0x7f884c108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:25.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 --2- 172.21.15.38:0/1528408738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c108340 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8840006c70 tx=0x7f8840006f70).stop 2024-02-20T13:16:25.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 shutdown_connections 2024-02-20T13:16:25.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 --2- 172.21.15.38:0/1528408738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f884c0faed0 0x7f884c108340 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:25.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 >> 172.21.15.38:0/1528408738 conn(0x7f884c0f6680 msgr2=0x7f884c0f8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:25.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 shutdown_connections 2024-02-20T13:16:25.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:25.339+0000 7f88537ec700 1 -- 172.21.15.38:0/1528408738 wait complete. 2024-02-20T13:16:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: audit 2024-02-20T13:16:25.336127+0000 mon.smithi038 (mon.0) 8 : audit [INF] from='client.? 172.21.15.38:0/1528408738' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T13:16:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: audit 2024-02-20T13:16:25.339514+0000 mon.smithi038 (mon.0) 9 : audit [INF] from='client.? 172.21.15.38:0/1528408738' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-20T13:16:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:Generating new minimal ceph.conf... 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f271970a700 1 Processor -- start 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f271970a700 1 -- start start 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f271970a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f27140fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f271970a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27140fcee0 con 0x7f27140fc5f0 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f2712ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f27140fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f2712ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f27140fc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53546/0 (socket says 172.21.15.38:53546) 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f2712ffd700 1 -- 172.21.15.38:0/1724526764 learned_addr learned my addr 172.21.15.38:0/1724526764 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f2712ffd700 1 -- 172.21.15.38:0/1724526764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140fd020 con 0x7f27140fc5f0 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.567+0000 7f2712ffd700 1 --2- 172.21.15.38:0/1724526764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f27140fc9f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f270400b5f0 tx=0x7f2704006ee0).ready entity=mon.0 client_cookie=e1321c2f3cebebb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:26.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f2711ffb700 1 -- 172.21.15.38:0/1724526764 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f270400d600 con 0x7f27140fc5f0 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f2711ffb700 1 -- 172.21.15.38:0/1724526764 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f270400dbc0 con 0x7f27140fc5f0 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f2711ffb700 1 -- 172.21.15.38:0/1724526764 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2704011b90 con 0x7f27140fc5f0 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 -- 172.21.15.38:0/1724526764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 msgr2=0x7f27140fc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 --2- 172.21.15.38:0/1724526764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f27140fc9f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f270400b5f0 tx=0x7f2704006ee0).stop 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 -- 172.21.15.38:0/1724526764 shutdown_connections 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 --2- 172.21.15.38:0/1724526764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f27140fc9f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 -- 172.21.15.38:0/1724526764 >> 172.21.15.38:0/1724526764 conn(0x7f27140f5e20 msgr2=0x7f27140f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 -- 172.21.15.38:0/1724526764 shutdown_connections 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.568+0000 7f271970a700 1 -- 172.21.15.38:0/1724526764 wait complete. 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f271970a700 1 Processor -- start 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f271970a700 1 -- start start 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f271970a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f271410cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:26.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f271970a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271410d450 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f2712ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f271410cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f2712ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f271410cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53554/0 (socket says 172.21.15.38:53554) 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.569+0000 7f2712ffd700 1 -- 172.21.15.38:0/1840266298 learned_addr learned my addr 172.21.15.38:0/1840266298 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f2712ffd700 1 -- 172.21.15.38:0/1840266298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270400b200 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f2712ffd700 1 --2- 172.21.15.38:0/1840266298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f271410cf60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2704000c00 tx=0x7f2704006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2704011560 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f270401b870 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271410d650 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f270401a690 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271410dae0 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f270402b070 con 0x7f27140fc5f0 2024-02-20T13:16:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.570+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2704024a70 con 0x7f27140fc5f0 2024-02-20T13:16:26.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.571+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2714045bb0 con 0x7f27140fc5f0 2024-02-20T13:16:26.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.573+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f270401f020 con 0x7f27140fc5f0 2024-02-20T13:16:26.641 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.640+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f27141069d0 con 0x7f27140fc5f0 2024-02-20T13:16:26.642 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.640+0000 7f26fb7fe700 1 -- 172.21.15.38:0/1840266298 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2704016070 con 0x7f27140fc5f0 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 msgr2=0x7f271410cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 --2- 172.21.15.38:0/1840266298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f271410cf60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2704000c00 tx=0x7f2704006ee0).stop 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 shutdown_connections 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 --2- 172.21.15.38:0/1840266298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27140fc5f0 0x7f271410cf60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 >> 172.21.15.38:0/1840266298 conn(0x7f27140f5e20 msgr2=0x7f27140f78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 shutdown_connections 2024-02-20T13:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:26.641+0000 7f271970a700 1 -- 172.21.15.38:0/1840266298 wait complete. 2024-02-20T13:16:26.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:26 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: audit 2024-02-20T13:16:26.641945+0000 mon.smithi038 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.38:0/1840266298' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:16:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:Restarting the monitor... 2024-02-20T13:16:27.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:27 smithi038 systemd[1]: Stopping Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715... 2024-02-20T13:16:27.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:27 smithi038 bash[38205]: Error: no container with name or ID "ceph-24090638-cff2-11ee-95bb-87774f69a715-mon.smithi038" found: no such container 2024-02-20T13:16:28.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: debug 2024-02-20T13:16:28.581+0000 7f99abed5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi038 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T13:16:28.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[37871]: debug 2024-02-20T13:16:28.581+0000 7f99abed5700 -1 mon.smithi038@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T13:16:32.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:31 smithi038 bash[38243]: ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038 2024-02-20T13:16:33.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:33 smithi038 systemd[1]: ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038.service: Succeeded. 2024-02-20T13:16:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:33 smithi038 systemd[1]: Stopped Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715. 2024-02-20T13:16:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:33 smithi038 systemd[1]: Starting Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715... 2024-02-20T13:16:37.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.099+0000 7f2bd758c700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T13:16:37.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.099+0000 7f2bd758c700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2024-02-20T13:16:37.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.099+0000 7f2bd758c700 0 pidfile_write: ignore empty --pid-file 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.103+0000 7f2bd758c700 0 load: jerasure load: lrc load: isa 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Compile date Jul 8 2021 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: DB SUMMARY 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: CURRENT file: CURRENT 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-20T13:16:37.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi038/store.db dir, Total Num: 1, files: 000004.sst 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi038/store.db: 000006.log size: 100467 ; 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.create_if_missing: 0 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T13:16:37.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.env: 0x56421aa531c0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.fs: Posix File System 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.info_log: 0x56421bc1fa40 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.statistics: (nil) 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.use_fsync: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T13:16:37.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.db_log_dir: 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi038/store.db 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T13:16:37.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.write_buffer_manager: 0x56421c896030 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.unordered_write: 0 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T13:16:37.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.row_cache: None 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.wal_filter: None 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T13:16:37.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.max_open_files: -1 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T13:16:37.455 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Compression algorithms supported: 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kZSTD supported: 0 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.104+0000 7f2bd758c700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000005 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.456 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.merge_operator: 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_filter: None 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56421bb38170) 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cache_index_and_filter_blocks: 1 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: pin_top_level_index_and_filter: 1 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: index_type: 0 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: data_block_index_type: 0 2024-02-20T13:16:37.457 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: index_shortening: 1 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: data_block_hash_table_util_ratio: 0.750000 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: hash_index_allow_collision: 1 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: checksum: 1 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: no_block_cache: 0 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_cache: 0x56421bb6ef10 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_cache_name: BinnedLRUCache 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_cache_options: 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: capacity : 536870912 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: num_shard_bits : 4 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: strict_capacity_limit : 0 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: high_pri_pool_ratio: 0.000 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_cache_compressed: (nil) 2024-02-20T13:16:37.458 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: persistent_cache: (nil) 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_size: 4096 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_size_deviation: 10 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_restart_interval: 16 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: index_block_restart_interval: 1 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: metadata_block_size: 4096 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: partition_filters: 0 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: use_delta_encoding: 1 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: whole_key_filtering: 1 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: verify_compression: 0 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: read_amp_bytes_per_bit: 0 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: format_version: 2 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: enable_index_compression: 1 2024-02-20T13:16:37.459 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: block_align: 0 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression: NoCompression 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.num_levels: 7 2024-02-20T13:16:37.460 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T13:16:37.461 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T13:16:37.462 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T13:16:37.463 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.ttl: 2592000 2024-02-20T13:16:37.464 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708434997107242, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.105+0000 7f2bd758c700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.106+0000 7f2bd758c700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.107+0000 7f2bd758c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708434997108356, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 95603, "table_properties": {"data_size": 93698, "index_size": 382, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12934, "raw_average_key_size": 49, "raw_value_size": 86578, "raw_average_value_size": 331, "num_data_blocks": 12, "num_entries": 261, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708434997, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.107+0000 7f2bd758c700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.107+0000 7f2bd758c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708434997108947, "job": 1, "event": "recovery_finished"} 2024-02-20T13:16:37.465 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.108+0000 7f2bd758c700 4 rocksdb: DB pointer 0x56421bc33800 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.108+0000 7f2bbe394700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.108+0000 7f2bbe394700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** DB Stats ** 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** Compaction Stats [default] ** 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:37.466 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: L0 2/0 95.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Sum 2/0 95.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** Compaction Stats [default] ** 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:16:37.467 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Cumulative compaction: 0.00 GB write, 24.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Interval compaction: 0.00 GB write, 24.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** Compaction Stats [default] ** 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: L0 2/0 95.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Sum 2/0 95.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T13:16:37.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** Compaction Stats [default] ** 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.6 0.00 0.00 1 0.001 0 0 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Cumulative compaction: 0.00 GB write, 24.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:37.469 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.109+0000 7f2bd758c700 0 starting mon.smithi038 rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi038 fsid 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 1 mon.smithi038@-1(???) e1 preinit fsid 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 0 mon.smithi038@-1(???).mds e1 new map 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 0 mon.smithi038@-1(???).mds e1 print_map 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: e1 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: legacy client fscid: -1 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.470 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: No filesystems configured 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 0 mon.smithi038@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 0 mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 0 mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 0 mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:26.641945+0000 mon.smithi038 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.38:0/1840266298' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:16:37.110+0000 7f2bd758c700 1 mon.smithi038@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:37.128068+0000 mon.smithi038 (mon.0) 1 : cluster [INF] mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:37.128128+0000 mon.smithi038 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:37.128162+0000 mon.smithi038 (mon.0) 3 : cluster [DBG] fsmap 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:37.128193+0000 mon.smithi038 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T13:16:37.471 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:37.128618+0000 mon.smithi038 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T13:16:37.684 INFO:teuthology.orchestra.run.smithi038.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-20T13:16:37.945 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 bash[38364]: a44dcaf5a38178cd7c848ba1d3458410792b1ec81871a8bbfabd4dd862c3b3b7 2024-02-20T13:16:37.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:37 smithi038 systemd[1]: Started Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715. 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.884+0000 7eff6836b700 1 Processor -- start 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.884+0000 7eff6836b700 1 -- start start 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.884+0000 7eff6836b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff600fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.884+0000 7eff6836b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff600fb7e0 con 0x7eff600faed0 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff66107700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff600fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff66107700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff600fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33512/0 (socket says 172.21.15.38:33512) 2024-02-20T13:16:39.890 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff66107700 1 -- 172.21.15.38:0/1231842680 learned_addr learned my addr 172.21.15.38:0/1231842680 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff66107700 1 -- 172.21.15.38:0/1231842680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff600fb920 con 0x7eff600faed0 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff66107700 1 --2- 172.21.15.38:0/1231842680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff600fb2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7eff5c00b5f0 tx=0x7eff5c006ee0).ready entity=mon.0 client_cookie=dcaf48f47d03a708 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff65105700 1 -- 172.21.15.38:0/1231842680 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c00d600 con 0x7eff600faed0 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff65105700 1 -- 172.21.15.38:0/1231842680 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7eff5c00dbc0 con 0x7eff600faed0 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.885+0000 7eff65105700 1 -- 172.21.15.38:0/1231842680 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c011b90 con 0x7eff600faed0 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 -- 172.21.15.38:0/1231842680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 msgr2=0x7eff600fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 --2- 172.21.15.38:0/1231842680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff600fb2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7eff5c00b5f0 tx=0x7eff5c006ee0).stop 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 -- 172.21.15.38:0/1231842680 shutdown_connections 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 --2- 172.21.15.38:0/1231842680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff600fb2f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 -- 172.21.15.38:0/1231842680 >> 172.21.15.38:0/1231842680 conn(0x7eff600f6680 msgr2=0x7eff600f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 -- 172.21.15.38:0/1231842680 shutdown_connections 2024-02-20T13:16:39.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.886+0000 7eff6836b700 1 -- 172.21.15.38:0/1231842680 wait complete. 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff6836b700 1 Processor -- start 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff6836b700 1 -- start start 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff6836b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff60101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff6836b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff60101f60 con 0x7eff600faed0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff66107700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff60101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff66107700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff60101a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33524/0 (socket says 172.21.15.38:33524) 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff66107700 1 -- 172.21.15.38:0/3920781066 learned_addr learned my addr 172.21.15.38:0/3920781066 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.887+0000 7eff66107700 1 -- 172.21.15.38:0/3920781066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c00b200 con 0x7eff600faed0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff66107700 1 --2- 172.21.15.38:0/3920781066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff60101a70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7eff5c006800 tx=0x7eff5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c00de50 con 0x7eff600faed0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff60102160 con 0x7eff600faed0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7eff5c01b870 con 0x7eff600faed0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c01a7a0 con 0x7eff600faed0 2024-02-20T13:16:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff601025f0 con 0x7eff600faed0 2024-02-20T13:16:39.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7eff5c00f070 con 0x7eff600faed0 2024-02-20T13:16:39.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.888+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7eff5c024c40 con 0x7eff600faed0 2024-02-20T13:16:39.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.889+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff60045bb0 con 0x7eff600faed0 2024-02-20T13:16:39.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.891+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7eff5c01f030 con 0x7eff600faed0 2024-02-20T13:16:39.959 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.957+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7eff600fcc10 con 0x7eff600faed0 2024-02-20T13:16:39.961 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.960+0000 7eff4effd700 1 -- 172.21.15.38:0/3920781066 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7eff5c0330c0 con 0x7eff600faed0 2024-02-20T13:16:39.962 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.960+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 msgr2=0x7eff60101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:39.963 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.961+0000 7eff6836b700 1 --2- 172.21.15.38:0/3920781066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff60101a70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7eff5c006800 tx=0x7eff5c006ee0).stop 2024-02-20T13:16:39.963 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.961+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 shutdown_connections 2024-02-20T13:16:39.963 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.961+0000 7eff6836b700 1 --2- 172.21.15.38:0/3920781066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff600faed0 0x7eff60101a70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:39.963 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.961+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 >> 172.21.15.38:0/3920781066 conn(0x7eff600f6680 msgr2=0x7eff6006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:39.963 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.961+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 shutdown_connections 2024-02-20T13:16:39.963 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:39.961+0000 7eff6836b700 1 -- 172.21.15.38:0/3920781066 wait complete. 2024-02-20T13:16:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-20T13:16:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-20T13:16:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:Creating mgr... 2024-02-20T13:16:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying port 9283 ... 2024-02-20T13:16:40.737 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to reset failed state of unit ceph-24090638-cff2-11ee-95bb-87774f69a715@mgr.smithi038.powepi.service: Unit ceph-24090638-cff2-11ee-95bb-87774f69a715@mgr.smithi038.powepi.service not loaded. 2024-02-20T13:16:40.743 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/ceph-24090638-cff2-11ee-95bb-87774f69a715.target.wants/ceph-24090638-cff2-11ee-95bb-87774f69a715@mgr.smithi038.powepi.service → /etc/systemd/system/ceph-24090638-cff2-11ee-95bb-87774f69a715@.service. 2024-02-20T13:16:41.399 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:39.961466+0000 mon.smithi038 (mon.0) 6 : audit [INF] from='client.? 172.21.15.38:0/3920781066' entity='client.admin' 2024-02-20T13:16:41.411 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2024-02-20T13:16:41.420 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2024-02-20T13:16:41.420 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2024-02-20T13:16:41.421 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T13:16:41.428 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2024-02-20T13:16:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2024-02-20T13:16:41.437 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2024-02-20T13:16:41.437 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-20T13:16:41.437 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr to start... 2024-02-20T13:16:41.437 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr... 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.982+0000 7f6f58b26700 1 Processor -- start 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.983+0000 7f6f58b26700 1 -- start start 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.983+0000 7f6f58b26700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f540fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.983+0000 7f6f58b26700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540fb7e0 con 0x7f6f540faed0 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.983+0000 7f6f5259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f540fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.983+0000 7f6f5259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f540fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33548/0 (socket says 172.21.15.38:33548) 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.983+0000 7f6f5259c700 1 -- 172.21.15.38:0/2617369259 learned_addr learned my addr 172.21.15.38:0/2617369259 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.984+0000 7f6f5259c700 1 -- 172.21.15.38:0/2617369259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f540fb920 con 0x7f6f540faed0 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.984+0000 7f6f5259c700 1 --2- 172.21.15.38:0/2617369259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f540fb2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6f4400b5f0 tx=0x7f6f44006ee0).ready entity=mon.0 client_cookie=3f21f0631dd32b52 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.984+0000 7f6f5159a700 1 -- 172.21.15.38:0/2617369259 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4400d600 con 0x7f6f540faed0 2024-02-20T13:16:41.985 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.984+0000 7f6f5159a700 1 -- 172.21.15.38:0/2617369259 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f6f4400dbc0 con 0x7f6f540faed0 2024-02-20T13:16:41.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.984+0000 7f6f5159a700 1 -- 172.21.15.38:0/2617369259 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f44011b90 con 0x7f6f540faed0 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.984+0000 7f6f58b26700 1 -- 172.21.15.38:0/2617369259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 msgr2=0x7f6f540fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 --2- 172.21.15.38:0/2617369259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f540fb2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6f4400b5f0 tx=0x7f6f44006ee0).stop 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 -- 172.21.15.38:0/2617369259 shutdown_connections 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 --2- 172.21.15.38:0/2617369259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f540fb2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 -- 172.21.15.38:0/2617369259 >> 172.21.15.38:0/2617369259 conn(0x7f6f540f6680 msgr2=0x7f6f540f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 -- 172.21.15.38:0/2617369259 shutdown_connections 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 -- 172.21.15.38:0/2617369259 wait complete. 2024-02-20T13:16:41.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 Processor -- start 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 -- start start 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f5410cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.985+0000 7f6f58b26700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5410d3d0 con 0x7f6f540faed0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f5259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f5410cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f5259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f5410cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33556/0 (socket says 172.21.15.38:33556) 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f5259c700 1 -- 172.21.15.38:0/344424638 learned_addr learned my addr 172.21.15.38:0/344424638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f5259c700 1 -- 172.21.15.38:0/344424638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4400b200 con 0x7f6f540faed0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f5259c700 1 --2- 172.21.15.38:0/344424638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f5410cee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6f44006800 tx=0x7f6f44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4400de50 con 0x7f6f540faed0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f6f4401b870 con 0x7f6f540faed0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5410d5d0 con 0x7f6f540faed0 2024-02-20T13:16:41.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4401a7a0 con 0x7f6f540faed0 2024-02-20T13:16:41.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5410da60 con 0x7f6f540faed0 2024-02-20T13:16:41.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.986+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f6f4400f070 con 0x7f6f540faed0 2024-02-20T13:16:41.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.987+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6f44024bf0 con 0x7f6f540faed0 2024-02-20T13:16:41.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.987+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f54045bb0 con 0x7f6f540faed0 2024-02-20T13:16:41.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:41.990+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6f4401f030 con 0x7f6f540faed0 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.059+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6f540fcc10 con 0x7f6f540faed0 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.059+0000 7f6f42ffd700 1 -- 172.21.15.38:0/344424638 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f6f44033090 con 0x7f6f540faed0 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "24090638-cff2-11ee-95bb-87774f69a715", 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2024-02-20T13:16:42.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T13:16:42.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T13:16:42.072 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.073 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": false, 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "nfs", 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2024-02-20T13:16:23.414658+0000", 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T13:16:42.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 msgr2=0x7f6f5410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 --2- 172.21.15.38:0/344424638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f5410cee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6f44006800 tx=0x7f6f44006ee0).stop 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 shutdown_connections 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 --2- 172.21.15.38:0/344424638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540faed0 0x7f6f5410cee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 >> 172.21.15.38:0/344424638 conn(0x7f6f540f6680 msgr2=0x7f6f540271f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 shutdown_connections 2024-02-20T13:16:42.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:42.061+0000 7f6f58b26700 1 -- 172.21.15.38:0/344424638 wait complete. 2024-02-20T13:16:42.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:42.059710+0000 mon.smithi038 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.38:0/344424638' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T13:16:42.734 INFO:teuthology.orchestra.run.smithi038.stderr:mgr not available, waiting (1/15)... 2024-02-20T13:16:45.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f83515a8700 1 Processor -- start 2024-02-20T13:16:45.299 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f83515a8700 1 -- start start 2024-02-20T13:16:45.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f83515a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c0febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:45.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f83515a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c0ff0e0 con 0x7f834c070cd0 2024-02-20T13:16:45.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f834affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c0febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:45.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f834affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c0febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33564/0 (socket says 172.21.15.38:33564) 2024-02-20T13:16:45.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f834affd700 1 -- 172.21.15.38:0/2225635824 learned_addr learned my addr 172.21.15.38:0/2225635824 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f834affd700 1 -- 172.21.15.38:0/2225635824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c0ff220 con 0x7f834c070cd0 2024-02-20T13:16:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.279+0000 7f834affd700 1 --2- 172.21.15.38:0/2225635824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c0febf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f833c00b5f0 tx=0x7f833c006ee0).ready entity=mon.0 client_cookie=36c94f8fd9e86b1f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f8349ffb700 1 -- 172.21.15.38:0/2225635824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833c00d600 con 0x7f834c070cd0 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f8349ffb700 1 -- 172.21.15.38:0/2225635824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f833c00dbc0 con 0x7f834c070cd0 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f8349ffb700 1 -- 172.21.15.38:0/2225635824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833c011b90 con 0x7f834c070cd0 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f83515a8700 1 -- 172.21.15.38:0/2225635824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 msgr2=0x7f834c0febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f83515a8700 1 --2- 172.21.15.38:0/2225635824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c0febf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f833c00b5f0 tx=0x7f833c006ee0).stop 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f83515a8700 1 -- 172.21.15.38:0/2225635824 shutdown_connections 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f83515a8700 1 --2- 172.21.15.38:0/2225635824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c0febf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.280+0000 7f83515a8700 1 -- 172.21.15.38:0/2225635824 >> 172.21.15.38:0/2225635824 conn(0x7f834c0f6690 msgr2=0x7f834c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.281+0000 7f83515a8700 1 -- 172.21.15.38:0/2225635824 shutdown_connections 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.281+0000 7f83515a8700 1 -- 172.21.15.38:0/2225635824 wait complete. 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.281+0000 7f83515a8700 1 Processor -- start 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.281+0000 7f83515a8700 1 -- start start 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.281+0000 7f83515a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c10ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.281+0000 7f83515a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c10b250 con 0x7f834c070cd0 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f834affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c10ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f834affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c10ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33576/0 (socket says 172.21.15.38:33576) 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f834affd700 1 -- 172.21.15.38:0/700670322 learned_addr learned my addr 172.21.15.38:0/700670322 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f834affd700 1 -- 172.21.15.38:0/700670322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c00b200 con 0x7f834c070cd0 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f834affd700 1 --2- 172.21.15.38:0/700670322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c10ad60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f833c006800 tx=0x7f833c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833c00de50 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f833c01b870 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c10b450 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f833c01a7a0 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.282+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c10c060 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.283+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f833c00f070 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.283+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f833c024c40 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.284+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c0591e0 con 0x7f834c070cd0 2024-02-20T13:16:45.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.286+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f833c01f030 con 0x7f834c070cd0 2024-02-20T13:16:45.359 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.356+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f834c0ffff0 con 0x7f834c070cd0 2024-02-20T13:16:45.359 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.357+0000 7f83337fe700 1 -- 172.21.15.38:0/700670322 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f833c033090 con 0x7f834c070cd0 2024-02-20T13:16:45.359 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "24090638-cff2-11ee-95bb-87774f69a715", 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2024-02-20T13:16:45.360 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T13:16:45.363 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T13:16:45.364 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": false, 2024-02-20T13:16:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "nfs", 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2024-02-20T13:16:23.414658+0000", 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:45.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 msgr2=0x7f834c10ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 --2- 172.21.15.38:0/700670322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c10ad60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f833c006800 tx=0x7f833c006ee0).stop 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 shutdown_connections 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 --2- 172.21.15.38:0/700670322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f834c070cd0 0x7f834c10ad60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 >> 172.21.15.38:0/700670322 conn(0x7f834c0f6690 msgr2=0x7f834c0f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 shutdown_connections 2024-02-20T13:16:45.369 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:45.358+0000 7f83515a8700 1 -- 172.21.15.38:0/700670322 wait complete. 2024-02-20T13:16:45.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:45.357409+0000 mon.smithi038 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.38:0/700670322' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T13:16:46.099 INFO:teuthology.orchestra.run.smithi038.stderr:mgr not available, waiting (2/15)... 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:46.637414+0000 mon.smithi038 (mon.0) 9 : cluster [INF] Activating manager daemon smithi038.powepi 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:46.640147+0000 mon.smithi038 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi038.powepi(active, starting, since 0.00286081s) 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.641137+0000 mon.smithi038 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.641349+0000 mon.smithi038 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.641523+0000 mon.smithi038 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.641714+0000 mon.smithi038 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.641904+0000 mon.smithi038 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mgr metadata", "who": "smithi038.powepi", "id": "smithi038.powepi"}]: dispatch 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:46.655716+0000 mon.smithi038 (mon.0) 16 : cluster [INF] Manager daemon smithi038.powepi is now available 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.667587+0000 mon.smithi038 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.669144+0000 mon.smithi038 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:16:46.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.670357+0000 mon.smithi038 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:16:46.947 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.675543+0000 mon.smithi038 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' 2024-02-20T13:16:46.947 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:46.685044+0000 mon.smithi038 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.38:0/2735512893' entity='mgr.smithi038.powepi' 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81fe4df700 1 Processor -- start 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81fe4df700 1 -- start start 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81fe4df700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f80febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81fe4df700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f80ff0d0 con 0x7f81f8027670 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f80febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f80febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38482/0 (socket says 172.21.15.38:38482) 2024-02-20T13:16:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.651+0000 7f81f7fff700 1 -- 172.21.15.38:0/2748025817 learned_addr learned my addr 172.21.15.38:0/2748025817 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.652+0000 7f81f7fff700 1 -- 172.21.15.38:0/2748025817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f80ff210 con 0x7f81f8027670 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.652+0000 7f81f7fff700 1 --2- 172.21.15.38:0/2748025817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f80febe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f81e800b5f0 tx=0x7f81e8006ee0).ready entity=mon.0 client_cookie=d9ff552cac68161c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.652+0000 7f81f6ffd700 1 -- 172.21.15.38:0/2748025817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e800d600 con 0x7f81f8027670 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.652+0000 7f81f6ffd700 1 -- 172.21.15.38:0/2748025817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f81e800dbc0 con 0x7f81f8027670 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.652+0000 7f81f6ffd700 1 -- 172.21.15.38:0/2748025817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e8011b90 con 0x7f81f8027670 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 -- 172.21.15.38:0/2748025817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 msgr2=0x7f81f80febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 --2- 172.21.15.38:0/2748025817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f80febe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f81e800b5f0 tx=0x7f81e8006ee0).stop 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 -- 172.21.15.38:0/2748025817 shutdown_connections 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 --2- 172.21.15.38:0/2748025817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f80febe0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 -- 172.21.15.38:0/2748025817 >> 172.21.15.38:0/2748025817 conn(0x7f81f80f6680 msgr2=0x7f81f80f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 -- 172.21.15.38:0/2748025817 shutdown_connections 2024-02-20T13:16:48.658 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 -- 172.21.15.38:0/2748025817 wait complete. 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 Processor -- start 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.653+0000 7f81fe4df700 1 -- start start 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81fe4df700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f810ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81fe4df700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f810b220 con 0x7f81f8027670 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f810ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f810ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38490/0 (socket says 172.21.15.38:38490) 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f7fff700 1 -- 172.21.15.38:0/1262158722 learned_addr learned my addr 172.21.15.38:0/1262158722 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f7fff700 1 -- 172.21.15.38:0/1262158722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e800b200 con 0x7f81f8027670 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f7fff700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f810ad30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f81e8006800 tx=0x7f81e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e800de50 con 0x7f81f8027670 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f81e801b870 con 0x7f81f8027670 2024-02-20T13:16:48.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e801a7a0 con 0x7f81f8027670 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.654+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f810b420 con 0x7f81f8027670 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.655+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f810c030 con 0x7f81f8027670 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.655+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 3) v1 ==== 37004+0+0 (secure 0 0 0) 0x7f81e800f070 con 0x7f81f8027670 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.655+0000 7f81f4ff9700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f81e4030f00 0x7f81e40333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.655+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f81e80429f0 con 0x7f81f8027670 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.656+0000 7f81f77fe700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f81e4030f00 0x7f81e40333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.656+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f80591e0 con 0x7f81f8027670 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.656+0000 7f81f77fe700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f81e4030f00 0x7f81e40333c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f81ec0096a0 tx=0x7f81ec006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:48.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.660+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f81e8016030 con 0x7f81f8027670 2024-02-20T13:16:48.885 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.882+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f81f80fffe0 con 0x7f81f8027670 2024-02-20T13:16:48.885 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.882+0000 7f81f4ff9700 1 -- 172.21.15.38:0/1262158722 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f81e8032090 con 0x7f81f8027670 2024-02-20T13:16:48.885 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.885 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:16:48.885 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "24090638-cff2-11ee-95bb-87774f69a715", 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T13:16:48.886 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:48.887 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T13:16:48.887 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T13:16:48.887 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.887 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T13:16:48.887 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:48.887 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T13:16:48.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T13:16:48.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": true, 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "nfs", 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T13:16:48.893 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2024-02-20T13:16:23.414658+0000", 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f81e4030f00 msgr2=0x7f81e40333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f81e4030f00 0x7f81e40333c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f81ec0096a0 tx=0x7f81ec006b40).stop 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 msgr2=0x7f81f810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f810ad30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f81e8006800 tx=0x7f81e8006ee0).stop 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 shutdown_connections 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f81e4030f00 0x7f81e40333c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 --2- 172.21.15.38:0/1262158722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81f8027670 0x7f81f810ad30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 >> 172.21.15.38:0/1262158722 conn(0x7f81f80f6680 msgr2=0x7f81f80f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 shutdown_connections 2024-02-20T13:16:48.894 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:48.884+0000 7f81fe4df700 1 -- 172.21.15.38:0/1262158722 wait complete. 2024-02-20T13:16:48.908 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:47.643621+0000 mon.smithi038 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi038.powepi(active, since 1.00633s) 2024-02-20T13:16:49.531 INFO:teuthology.orchestra.run.smithi038.stderr:mgr is available 2024-02-20T13:16:49.804 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:48.883101+0000 mon.smithi038 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.38:0/1262158722' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T13:16:50.083 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.079+0000 7f7790ce0700 1 Processor -- start 2024-02-20T13:16:50.083 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f7790ce0700 1 -- start start 2024-02-20T13:16:50.083 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f7790ce0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f7790ce0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0ff0d0 con 0x7f778c027670 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f778a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f778a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38496/0 (socket says 172.21.15.38:38496) 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f778a59c700 1 -- 172.21.15.38:0/1137546441 learned_addr learned my addr 172.21.15.38:0/1137546441 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f778a59c700 1 -- 172.21.15.38:0/1137546441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0ff210 con 0x7f778c027670 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.080+0000 7f778a59c700 1 --2- 172.21.15.38:0/1137546441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c0febe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f777c00b5f0 tx=0x7f777c006ee0).ready entity=mon.0 client_cookie=80981c0602517b2f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f778959a700 1 -- 172.21.15.38:0/1137546441 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777c00d600 con 0x7f778c027670 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f778959a700 1 -- 172.21.15.38:0/1137546441 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f777c00dbc0 con 0x7f778c027670 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f778959a700 1 -- 172.21.15.38:0/1137546441 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777c011b90 con 0x7f778c027670 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 -- 172.21.15.38:0/1137546441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 msgr2=0x7f778c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 --2- 172.21.15.38:0/1137546441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c0febe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f777c00b5f0 tx=0x7f777c006ee0).stop 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 -- 172.21.15.38:0/1137546441 shutdown_connections 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 --2- 172.21.15.38:0/1137546441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c0febe0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:50.084 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 -- 172.21.15.38:0/1137546441 >> 172.21.15.38:0/1137546441 conn(0x7f778c0f6680 msgr2=0x7f778c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 -- 172.21.15.38:0/1137546441 shutdown_connections 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.081+0000 7f7790ce0700 1 -- 172.21.15.38:0/1137546441 wait complete. 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f7790ce0700 1 Processor -- start 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f7790ce0700 1 -- start start 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f7790ce0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f7790ce0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c10d3b0 con 0x7f778c027670 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f778a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f778a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38502/0 (socket says 172.21.15.38:38502) 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.082+0000 7f778a59c700 1 -- 172.21.15.38:0/1041838476 learned_addr learned my addr 172.21.15.38:0/1041838476 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:50.085 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f778a59c700 1 -- 172.21.15.38:0/1041838476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c00b200 con 0x7f778c027670 2024-02-20T13:16:50.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f778a59c700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c10cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f777c006800 tx=0x7f777c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:50.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777c00de50 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f777c01b870 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777c01a7a0 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c10d5b0 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.083+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c10da40 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.084+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f777c00f070 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.084+0000 7f7782ffd700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f776c030fa0 0x7f776c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.084+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f777c042a80 con 0x7f778c027670 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.084+0000 7f7789d9b700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f776c030fa0 0x7f776c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.084+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c045bb0 con 0x7f778c027670 2024-02-20T13:16:50.088 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.084+0000 7f7789d9b700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f776c030fa0 0x7f776c033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f77740096a0 tx=0x7f7774006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:50.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.089+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f777c016030 con 0x7f778c027670 2024-02-20T13:16:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.247+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f778c107010 con 0x7f778c027670 2024-02-20T13:16:50.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.248+0000 7f7782ffd700 1 -- 172.21.15.38:0/1041838476 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f777c032020 con 0x7f778c027670 2024-02-20T13:16:50.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [global] 2024-02-20T13:16:50.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: fsid = 24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:16:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-20T13:16:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-20T13:16:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-20T13:16:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-20T13:16:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-20T13:16:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [mgr] 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [osd] 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-20T13:16:50.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f776c030fa0 msgr2=0x7f776c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f776c030fa0 0x7f776c033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f77740096a0 tx=0x7f7774006b40).stop 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 msgr2=0x7f778c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c10cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f777c006800 tx=0x7f777c006ee0).stop 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 shutdown_connections 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f776c030fa0 0x7f776c033460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 --2- 172.21.15.38:0/1041838476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f778c027670 0x7f778c10cec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 >> 172.21.15.38:0/1041838476 conn(0x7f778c0f6680 msgr2=0x7f778c0f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 shutdown_connections 2024-02-20T13:16:50.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:50.250+0000 7f7790ce0700 1 -- 172.21.15.38:0/1041838476 wait complete. 2024-02-20T13:16:50.829 INFO:teuthology.orchestra.run.smithi038.stderr:Enabling cephadm module... 2024-02-20T13:16:50.831 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:50 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:49.647940+0000 mon.smithi038 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi038.powepi(active, since 3s) 2024-02-20T13:16:50.831 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:50 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:50.248399+0000 mon.smithi038 (mon.0) 25 : audit [INF] from='client.? 172.21.15.38:0/1041838476' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T13:16:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.375+0000 7f4a35de1700 1 Processor -- start 2024-02-20T13:16:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.375+0000 7f4a35de1700 1 -- start start 2024-02-20T13:16:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a35de1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a30070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a35de1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30072060 con 0x7f4a30071c40 2024-02-20T13:16:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a30070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a30070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38512/0 (socket says 172.21.15.38:38512) 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2f7fe700 1 -- 172.21.15.38:0/2884858810 learned_addr learned my addr 172.21.15.38:0/2884858810 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2f7fe700 1 -- 172.21.15.38:0/2884858810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30070630 con 0x7f4a30071c40 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2f7fe700 1 --2- 172.21.15.38:0/2884858810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a30070140 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4a2000b5f0 tx=0x7f4a20006ee0).ready entity=mon.0 client_cookie=3a719eed092bf6c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2e7fc700 1 -- 172.21.15.38:0/2884858810 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2000d600 con 0x7f4a30071c40 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2e7fc700 1 -- 172.21.15.38:0/2884858810 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4a2000dbc0 con 0x7f4a30071c40 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.376+0000 7f4a2e7fc700 1 -- 172.21.15.38:0/2884858810 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a20011b90 con 0x7f4a30071c40 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 -- 172.21.15.38:0/2884858810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 msgr2=0x7f4a30070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 --2- 172.21.15.38:0/2884858810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a30070140 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4a2000b5f0 tx=0x7f4a20006ee0).stop 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 -- 172.21.15.38:0/2884858810 shutdown_connections 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 --2- 172.21.15.38:0/2884858810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a30070140 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 -- 172.21.15.38:0/2884858810 >> 172.21.15.38:0/2884858810 conn(0x7f4a300f7790 msgr2=0x7f4a300f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 -- 172.21.15.38:0/2884858810 shutdown_connections 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.377+0000 7f4a35de1700 1 -- 172.21.15.38:0/2884858810 wait complete. 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a35de1700 1 Processor -- start 2024-02-20T13:16:51.382 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a35de1700 1 -- start start 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a35de1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a3010cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a35de1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3010d420 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a3010cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a3010cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38520/0 (socket says 172.21.15.38:38520) 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a2f7fe700 1 -- 172.21.15.38:0/608060668 learned_addr learned my addr 172.21.15.38:0/608060668 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.378+0000 7f4a2f7fe700 1 -- 172.21.15.38:0/608060668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2000b200 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.379+0000 7f4a2f7fe700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a3010cf30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4a2000c010 tx=0x7f4a20006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.379+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2000de90 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.379+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4a2001b870 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.379+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3010d620 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.379+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2001a670 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.379+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3010dab0 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.380+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f4a2000f070 con 0x7f4a30071c40 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.380+0000 7f4a27fff700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f4a10030fa0 0x7f4a10033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:51.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.380+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4a200429a0 con 0x7f4a30071c40 2024-02-20T13:16:51.384 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.380+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a30045bb0 con 0x7f4a30071c40 2024-02-20T13:16:51.384 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.380+0000 7f4a2effd700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f4a10030fa0 0x7f4a10033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:51.384 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.380+0000 7f4a2effd700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f4a10030fa0 0x7f4a10033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4a180096a0 tx=0x7f4a18006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:51.385 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.384+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4a20016030 con 0x7f4a30071c40 2024-02-20T13:16:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.586+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4a301021a0 con 0x7f4a30071c40 2024-02-20T13:16:51.659 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.658+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f4a20024410 con 0x7f4a30071c40 2024-02-20T13:16:51.660 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.660+0000 7f4a27fff700 1 -- 172.21.15.38:0/608060668 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4a20019950 con 0x7f4a30071c40 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.661+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f4a10030fa0 msgr2=0x7f4a10033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.661+0000 7f4a35de1700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f4a10030fa0 0x7f4a10033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4a180096a0 tx=0x7f4a18006b40).stop 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.661+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 msgr2=0x7f4a3010cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.661+0000 7f4a35de1700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a3010cf30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4a2000c010 tx=0x7f4a20006ee0).stop 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.662+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 shutdown_connections 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.662+0000 7f4a35de1700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f4a10030fa0 0x7f4a10033460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.662+0000 7f4a35de1700 1 --2- 172.21.15.38:0/608060668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a30071c40 0x7f4a3010cf30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.662+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 >> 172.21.15.38:0/608060668 conn(0x7f4a300f7790 msgr2=0x7f4a300f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.662+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 shutdown_connections 2024-02-20T13:16:51.663 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:51.662+0000 7f4a35de1700 1 -- 172.21.15.38:0/608060668 wait complete. 2024-02-20T13:16:51.945 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:51.587404+0000 mon.smithi038 (mon.0) 26 : audit [INF] from='client.? 172.21.15.38:0/608060668' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.737+0000 7f2c9cd58700 1 Processor -- start 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.737+0000 7f2c9cd58700 1 -- start start 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.738+0000 7f2c9cd58700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c98070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.738+0000 7f2c9cd58700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98072080 con 0x7f2c98071c60 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.738+0000 7f2c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c98070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.738+0000 7f2c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c98070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38538/0 (socket says 172.21.15.38:38538) 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.738+0000 7f2c9659c700 1 -- 172.21.15.38:0/3979340490 learned_addr learned my addr 172.21.15.38:0/3979340490 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.739+0000 7f2c9659c700 1 -- 172.21.15.38:0/3979340490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98070650 con 0x7f2c98071c60 2024-02-20T13:16:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.739+0000 7f2c9659c700 1 --2- 172.21.15.38:0/3979340490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c98070160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f2c8800b5f0 tx=0x7f2c88006ee0).ready entity=mon.0 client_cookie=68367c9a0186514 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.739+0000 7f2c9559a700 1 -- 172.21.15.38:0/3979340490 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8800d600 con 0x7f2c98071c60 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.739+0000 7f2c9559a700 1 -- 172.21.15.38:0/3979340490 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f2c8800dbc0 con 0x7f2c98071c60 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.739+0000 7f2c9559a700 1 -- 172.21.15.38:0/3979340490 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c88011b90 con 0x7f2c98071c60 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 -- 172.21.15.38:0/3979340490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 msgr2=0x7f2c98070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 --2- 172.21.15.38:0/3979340490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c98070160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f2c8800b5f0 tx=0x7f2c88006ee0).stop 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 -- 172.21.15.38:0/3979340490 shutdown_connections 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 --2- 172.21.15.38:0/3979340490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c98070160 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 -- 172.21.15.38:0/3979340490 >> 172.21.15.38:0/3979340490 conn(0x7f2c980f7770 msgr2=0x7f2c980f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 -- 172.21.15.38:0/3979340490 shutdown_connections 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.740+0000 7f2c9cd58700 1 -- 172.21.15.38:0/3979340490 wait complete. 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9cd58700 1 Processor -- start 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9cd58700 1 -- start start 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9cd58700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c9810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9cd58700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9810d3c0 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c9810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c9810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38550/0 (socket says 172.21.15.38:38550) 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9659c700 1 -- 172.21.15.38:0/1580787558 learned_addr learned my addr 172.21.15.38:0/1580787558 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9659c700 1 -- 172.21.15.38:0/1580787558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8800b200 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c9659c700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c9810ced0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f2c8800e040 tx=0x7f2c88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.741+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8801b570 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.742+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9810d5c0 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.742+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f2c8801bb30 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.742+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8801a820 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.742+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9810da50 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.742+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f2c8802b070 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.743+0000 7f2c86ffd700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 0x7f2c7c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.743+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2c88042d80 con 0x7f2c98071c60 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.743+0000 7f2c95d9b700 1 -- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 msgr2=0x7f2c7c033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2401164108 2024-02-20T13:16:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.743+0000 7f2c95d9b700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 0x7f2c7c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:16:52.760 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.743+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c98045bb0 con 0x7f2c98071c60 2024-02-20T13:16:52.760 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.751+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2c88016020 con 0x7f2c98071c60 2024-02-20T13:16:52.945 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.944+0000 7f2c95d9b700 1 -- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 msgr2=0x7f2c7c033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2401164108 2024-02-20T13:16:52.945 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.944+0000 7f2c95d9b700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 0x7f2c7c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:16:52.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:51.660326+0000 mon.smithi038 (mon.0) 27 : audit [INF] from='client.? 172.21.15.38:0/608060668' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-20T13:16:52.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:51.660414+0000 mon.smithi038 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi038.powepi(active, since 5s) 2024-02-20T13:16:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.988+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2c981069a0 con 0x7f2c98071c60 2024-02-20T13:16:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.989+0000 7f2c86ffd700 1 -- 172.21.15.38:0/1580787558 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f2c88016030 con 0x7f2c98071c60 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": true, 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "active_name": "smithi038.powepi", 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 msgr2=0x7f2c7c033460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 0x7f2c7c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:16:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 msgr2=0x7f2c9810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c9810ced0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f2c8800e040 tx=0x7f2c88006ee0).stop 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 shutdown_connections 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f2c7c030fa0 0x7f2c7c033460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 --2- 172.21.15.38:0/1580787558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c98071c60 0x7f2c9810ced0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.992+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 >> 172.21.15.38:0/1580787558 conn(0x7f2c980f7770 msgr2=0x7f2c980f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.993+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 shutdown_connections 2024-02-20T13:16:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:52.993+0000 7f2c9cd58700 1 -- 172.21.15.38:0/1580787558 wait complete. 2024-02-20T13:16:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for the mgr to restart... 2024-02-20T13:16:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr epoch 5... 2024-02-20T13:16:53.825 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:52.989460+0000 mon.smithi038 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.38:0/1580787558' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T13:16:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c2d98700 1 Processor -- start 2024-02-20T13:16:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c2d98700 1 -- start start 2024-02-20T13:16:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c2d98700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc0fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c2d98700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc0fb7e0 con 0x7f25bc0faed0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c0b34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc0fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c0b34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc0fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38556/0 (socket says 172.21.15.38:38556) 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c0b34700 1 -- 172.21.15.38:0/3807607412 learned_addr learned my addr 172.21.15.38:0/3807607412 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c0b34700 1 -- 172.21.15.38:0/3807607412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0fb920 con 0x7f25bc0faed0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.150+0000 7f25c0b34700 1 --2- 172.21.15.38:0/3807607412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc0fb2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f25b000b870 tx=0x7f25b0006f70).ready entity=mon.0 client_cookie=1003564fd6223534 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25bb7fe700 1 -- 172.21.15.38:0/3807607412 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25b000da20 con 0x7f25bc0faed0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25bb7fe700 1 -- 172.21.15.38:0/3807607412 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f25b0010470 con 0x7f25bc0faed0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25bb7fe700 1 -- 172.21.15.38:0/3807607412 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25b000f5b0 con 0x7f25bc0faed0 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 -- 172.21.15.38:0/3807607412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 msgr2=0x7f25bc0fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 --2- 172.21.15.38:0/3807607412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc0fb2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f25b000b870 tx=0x7f25b0006f70).stop 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 -- 172.21.15.38:0/3807607412 shutdown_connections 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 --2- 172.21.15.38:0/3807607412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc0fb2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 -- 172.21.15.38:0/3807607412 >> 172.21.15.38:0/3807607412 conn(0x7f25bc0f6680 msgr2=0x7f25bc0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 -- 172.21.15.38:0/3807607412 shutdown_connections 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.151+0000 7f25c2d98700 1 -- 172.21.15.38:0/3807607412 wait complete. 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.152+0000 7f25c2d98700 1 Processor -- start 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.152+0000 7f25c2d98700 1 -- start start 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.152+0000 7f25c2d98700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc108b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.152+0000 7f25c2d98700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc109020 con 0x7f25bc0faed0 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.152+0000 7f25c0b34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc108b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25c0b34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc108b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38572/0 (socket says 172.21.15.38:38572) 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25c0b34700 1 -- 172.21.15.38:0/2677421351 learned_addr learned my addr 172.21.15.38:0/2677421351 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25c0b34700 1 -- 172.21.15.38:0/2677421351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b000b480 con 0x7f25bc0faed0 2024-02-20T13:16:54.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25c0b34700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc108b30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f25b0000c00 tx=0x7f25b0006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:54.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25b000f3d0 con 0x7f25bc0faed0 2024-02-20T13:16:54.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f25b000fe90 con 0x7f25bc0faed0 2024-02-20T13:16:54.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc109220 con 0x7f25bc0faed0 2024-02-20T13:16:54.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc1096b0 con 0x7f25bc0faed0 2024-02-20T13:16:54.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.153+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25b0019b80 con 0x7f25bc0faed0 2024-02-20T13:16:54.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.154+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f25b0016030 con 0x7f25bc0faed0 2024-02-20T13:16:54.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.154+0000 7f25b97fa700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:54.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.154+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f25b0045b40 con 0x7f25bc0faed0 2024-02-20T13:16:54.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.154+0000 7f25bbfff700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 msgr2=0x7f25a0033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2401164108 2024-02-20T13:16:54.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.155+0000 7f25bbfff700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:16:54.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.155+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 --> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25bc0591e0 con 0x7f25a0030fa0 2024-02-20T13:16:54.356 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.356+0000 7f25bbfff700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 msgr2=0x7f25a0033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2401164108 2024-02-20T13:16:54.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.356+0000 7f25bbfff700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:16:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.757+0000 7f25bbfff700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 msgr2=0x7f25a0033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2401164108 2024-02-20T13:16:54.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:54.757+0000 7f25bbfff700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T13:16:55.558 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:55.557+0000 7f25bbfff700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 msgr2=0x7f25a0033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2401164108 2024-02-20T13:16:55.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:55.557+0000 7f25bbfff700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T13:16:57.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:57.050+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) v1 ==== 36945+0+0 (secure 0 0 0) 0x7f25b002da00 con 0x7f25bc0faed0 2024-02-20T13:16:57.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:57.050+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 msgr2=0x7f25a0033460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:16:57.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:57.050+0000 7f25b97fa700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:16:57.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:56.999173+0000 mon.smithi038 (mon.0) 30 : cluster [INF] Active manager daemon smithi038.powepi restarted 2024-02-20T13:16:57.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:56.999566+0000 mon.smithi038 (mon.0) 31 : cluster [INF] Activating manager daemon smithi038.powepi 2024-02-20T13:16:57.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:57.001606+0000 mon.smithi038 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-20T13:16:58.056 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.054+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f25b002d440 con 0x7f25bc0faed0 2024-02-20T13:16:58.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.054+0000 7f25b97fa700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f25a0034850 0x7f25a0036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:58.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.054+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25bc0591e0 con 0x7f25a0034850 2024-02-20T13:16:58.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.054+0000 7f25bbfff700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f25a0034850 0x7f25a0036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:58.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.055+0000 7f25bbfff700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f25a0034850 0x7f25a0036c40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f25a80036d0 tx=0x7f25a8006ca0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:58.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.056+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f25bc0591e0 con 0x7f25a0034850 2024-02-20T13:16:58.065 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.063+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f25bc045bb0 con 0x7f25a0034850 2024-02-20T13:16:58.065 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25b97fa700 1 -- 172.21.15.38:0/2677421351 <== mgr.14116 v2:172.21.15.38:6800/4148388115 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f25bc045bb0 con 0x7f25a0034850 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f25a0034850 msgr2=0x7f25a0036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f25a0034850 0x7f25a0036c40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f25a80036d0 tx=0x7f25a8006ca0).stop 2024-02-20T13:16:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 msgr2=0x7f25bc108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc108b30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f25b0000c00 tx=0x7f25b0006f70).stop 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 shutdown_connections 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f25a0034850 0x7f25a0036c40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25bc0faed0 0x7f25bc108b30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 --2- 172.21.15.38:0/2677421351 >> [v2:172.21.15.38:6800/2401164108,v1:172.21.15.38:6801/2401164108] conn(0x7f25a0030fa0 0x7f25a0033460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 >> 172.21.15.38:0/2677421351 conn(0x7f25bc0f6680 msgr2=0x7f25bc0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.064+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 shutdown_connections 2024-02-20T13:16:58.068 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:58.065+0000 7f25c2d98700 1 -- 172.21.15.38:0/2677421351 wait complete. 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:57.052407+0000 mon.smithi038 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi038.powepi(active, starting, since 0.0529144s) 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.054962+0000 mon.smithi038 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.055575+0000 mon.smithi038 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mgr metadata", "who": "smithi038.powepi", "id": "smithi038.powepi"}]: dispatch 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.056439+0000 mon.smithi038 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.056585+0000 mon.smithi038 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.056707+0000 mon.smithi038 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:57.070787+0000 mon.smithi038 (mon.0) 39 : cluster [INF] Manager daemon smithi038.powepi is now available 2024-02-20T13:16:58.439 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.084254+0000 mon.smithi038 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.087015+0000 mon.smithi038 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.089556+0000 mon.smithi038 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.184454+0000 mon.smithi038 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.185166+0000 mon.smithi038 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.186581+0000 mon.smithi038 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.187963+0000 mon.smithi038 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.199365+0000 mon.smithi038 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:16:58.440 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:57.200501+0000 mon.smithi038 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:16:58.708 INFO:teuthology.orchestra.run.smithi038.stderr:mgr epoch 5 is available 2024-02-20T13:16:58.708 INFO:teuthology.orchestra.run.smithi038.stderr:Setting orchestrator backend to cephadm... 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.286+0000 7f83b86ce700 1 Processor -- start 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.286+0000 7f83b86ce700 1 -- start start 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b86ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b00fdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b86ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00fdf90 con 0x7f83b00fd680 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b646a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b00fdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b646a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b00fdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44494/0 (socket says 172.21.15.38:44494) 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b646a700 1 -- 172.21.15.38:0/2088588413 learned_addr learned my addr 172.21.15.38:0/2088588413 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b646a700 1 -- 172.21.15.38:0/2088588413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00fe0d0 con 0x7f83b00fd680 2024-02-20T13:16:59.291 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b646a700 1 --2- 172.21.15.38:0/2088588413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b00fdaa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f83ac00b5f0 tx=0x7f83ac006ee0).ready entity=mon.0 client_cookie=3888293584730740 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:59.292 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b5468700 1 -- 172.21.15.38:0/2088588413 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac00d600 con 0x7f83b00fd680 2024-02-20T13:16:59.292 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b5468700 1 -- 172.21.15.38:0/2088588413 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f83ac00dbc0 con 0x7f83b00fd680 2024-02-20T13:16:59.292 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.287+0000 7f83b5468700 1 -- 172.21.15.38:0/2088588413 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac011b90 con 0x7f83b00fd680 2024-02-20T13:16:59.292 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 -- 172.21.15.38:0/2088588413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 msgr2=0x7f83b00fdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 --2- 172.21.15.38:0/2088588413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b00fdaa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f83ac00b5f0 tx=0x7f83ac006ee0).stop 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 -- 172.21.15.38:0/2088588413 shutdown_connections 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 --2- 172.21.15.38:0/2088588413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b00fdaa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 -- 172.21.15.38:0/2088588413 >> 172.21.15.38:0/2088588413 conn(0x7f83b00f5e20 msgr2=0x7f83b00f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 -- 172.21.15.38:0/2088588413 shutdown_connections 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.288+0000 7f83b86ce700 1 -- 172.21.15.38:0/2088588413 wait complete. 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b86ce700 1 Processor -- start 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b86ce700 1 -- start start 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b86ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b010ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b86ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b010b280 con 0x7f83b00fd680 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b646a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b010ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b646a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b010ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44502/0 (socket says 172.21.15.38:44502) 2024-02-20T13:16:59.293 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b646a700 1 -- 172.21.15.38:0/390036521 learned_addr learned my addr 172.21.15.38:0/390036521 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:16:59.294 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b646a700 1 -- 172.21.15.38:0/390036521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac00b200 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.289+0000 7f83b646a700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b010ad90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f83ac000c00 tx=0x7f83ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.290+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac01b410 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.290+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f83ac01b9d0 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.290+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b010b480 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.290+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac01a620 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.290+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b010c090 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.290+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f83ac02b070 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.291+0000 7f83a2ffd700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8398030ff0 0x7f83980334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.291+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b00591e0 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.291+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f83ac042b50 con 0x7f83b00fd680 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.291+0000 7f83b5c69700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8398030ff0 0x7f83980334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:16:59.295 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.291+0000 7f83b5c69700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8398030ff0 0x7f83980334b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f83a40096a0 tx=0x7f83a4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:16:59.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.295+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f83ac016080 con 0x7f83b00fd680 2024-02-20T13:16:59.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:58.057268+0000 mgr.smithi038.powepi (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T13:16:59.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:58.058239+0000 mon.smithi038 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi038.powepi(active, since 1.05874s) 2024-02-20T13:16:59.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:16:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:58.065449+0000 mgr.smithi038.powepi (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T13:16:59.466 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.464+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b010c390 con 0x7f8398030ff0 2024-02-20T13:16:59.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.469+0000 7f83a2ffd700 1 -- 172.21.15.38:0/390036521 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f83b010c390 con 0x7f8398030ff0 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.471+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8398030ff0 msgr2=0x7f83980334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8398030ff0 0x7f83980334b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f83a40096a0 tx=0x7f83a4006b40).stop 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 msgr2=0x7f83b010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b010ad90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f83ac000c00 tx=0x7f83ac006ee0).stop 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 shutdown_connections 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8398030ff0 0x7f83980334b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:59.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 --2- 172.21.15.38:0/390036521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83b00fd680 0x7f83b010ad90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:16:59.475 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 >> 172.21.15.38:0/390036521 conn(0x7f83b00f5e20 msgr2=0x7f83b00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:16:59.475 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 shutdown_connections 2024-02-20T13:16:59.475 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:16:59.472+0000 7f83b86ce700 1 -- 172.21.15.38:0/390036521 wait complete. 2024-02-20T13:17:00.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:16:59.070296+0000 mon.smithi038 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi038.powepi(active, since 2s) 2024-02-20T13:17:00.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:59.466582+0000 mgr.smithi038.powepi (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:00.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:59.470629+0000 mon.smithi038 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:00.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:16:59.474309+0000 mon.smithi038 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba25b33700 1 Processor -- start 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba25b33700 1 -- start start 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba25b33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba200fb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba25b33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba200fbc70 con 0x7fba200fb360 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba200fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba200fb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44512/0 (socket says 172.21.15.38:44512) 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba1f7fe700 1 -- 172.21.15.38:0/2617202378 learned_addr learned my addr 172.21.15.38:0/2617202378 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.512+0000 7fba1f7fe700 1 -- 172.21.15.38:0/2617202378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba200fbdb0 con 0x7fba200fb360 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.513+0000 7fba1f7fe700 1 --2- 172.21.15.38:0/2617202378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba200fb780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fba10009960 tx=0x7fba100092d0).ready entity=mon.0 client_cookie=47e1d690d8154799 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.513+0000 7fba1e7fc700 1 -- 172.21.15.38:0/2617202378 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1000b9c0 con 0x7fba200fb360 2024-02-20T13:17:00.515 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.513+0000 7fba1e7fc700 1 -- 172.21.15.38:0/2617202378 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fba10011400 con 0x7fba200fb360 2024-02-20T13:17:00.516 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.513+0000 7fba1e7fc700 1 -- 172.21.15.38:0/2617202378 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba10010580 con 0x7fba200fb360 2024-02-20T13:17:00.516 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.513+0000 7fba25b33700 1 -- 172.21.15.38:0/2617202378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 msgr2=0x7fba200fb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:00.516 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.513+0000 7fba25b33700 1 --2- 172.21.15.38:0/2617202378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba200fb780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fba10009960 tx=0x7fba100092d0).stop 2024-02-20T13:17:00.517 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.514+0000 7fba25b33700 1 -- 172.21.15.38:0/2617202378 shutdown_connections 2024-02-20T13:17:00.517 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.514+0000 7fba25b33700 1 --2- 172.21.15.38:0/2617202378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba200fb780 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:00.517 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.514+0000 7fba25b33700 1 -- 172.21.15.38:0/2617202378 >> 172.21.15.38:0/2617202378 conn(0x7fba200f5df0 msgr2=0x7fba200f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.514+0000 7fba25b33700 1 -- 172.21.15.38:0/2617202378 shutdown_connections 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.514+0000 7fba25b33700 1 -- 172.21.15.38:0/2617202378 wait complete. 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.514+0000 7fba25b33700 1 Processor -- start 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba25b33700 1 -- start start 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba25b33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba201112f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba25b33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba201117e0 con 0x7fba200fb360 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba201112f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba201112f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44514/0 (socket says 172.21.15.38:44514) 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba1f7fe700 1 -- 172.21.15.38:0/3317143292 learned_addr learned my addr 172.21.15.38:0/3317143292 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba1f7fe700 1 -- 172.21.15.38:0/3317143292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba10009570 con 0x7fba200fb360 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba1f7fe700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba201112f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fba10000f80 tx=0x7fba100092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.515+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba100107f0 con 0x7fba200fb360 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.516+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fba10010db0 con 0x7fba200fb360 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.516+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba201119e0 con 0x7fba200fb360 2024-02-20T13:17:00.518 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.516+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1001bd30 con 0x7fba200fb360 2024-02-20T13:17:00.519 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.516+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba20111e70 con 0x7fba200fb360 2024-02-20T13:17:00.519 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.517+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fba1000f070 con 0x7fba200fb360 2024-02-20T13:17:00.519 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.517+0000 7fba17fff700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fba00030fa0 0x7fba00033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:00.520 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.517+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20045bb0 con 0x7fba200fb360 2024-02-20T13:17:00.520 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.517+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fba10042f60 con 0x7fba200fb360 2024-02-20T13:17:00.520 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.517+0000 7fba1effd700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fba00030fa0 0x7fba00033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:00.520 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.517+0000 7fba1effd700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fba00030fa0 0x7fba00033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fba080096a0 tx=0x7fba08006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:00.523 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.521+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fba10011e60 con 0x7fba200fb360 2024-02-20T13:17:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.681+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fba201126d0 con 0x7fba00030fa0 2024-02-20T13:17:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.682+0000 7fba17fff700 1 -- 172.21.15.38:0/3317143292 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fba201126d0 con 0x7fba00030fa0 2024-02-20T13:17:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: value unchanged 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.683+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fba00030fa0 msgr2=0x7fba00033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.683+0000 7fba25b33700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fba00030fa0 0x7fba00033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fba080096a0 tx=0x7fba08006b40).stop 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.683+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 msgr2=0x7fba201112f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.683+0000 7fba25b33700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba201112f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fba10000f80 tx=0x7fba100092d0).stop 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.684+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 shutdown_connections 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.684+0000 7fba25b33700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fba00030fa0 0x7fba00033460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.684+0000 7fba25b33700 1 --2- 172.21.15.38:0/3317143292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba200fb360 0x7fba201112f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.684+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 >> 172.21.15.38:0/3317143292 conn(0x7fba200f5df0 msgr2=0x7fba200f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.684+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 shutdown_connections 2024-02-20T13:17:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:00.684+0000 7fba25b33700 1 -- 172.21.15.38:0/3317143292 wait complete. 2024-02-20T13:17:01.271 INFO:teuthology.orchestra.run.smithi038.stderr:Generating ssh key... 2024-02-20T13:17:01.273 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:00.683493+0000 mgr.smithi038.powepi (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.772+0000 7f31aa1c1700 1 Processor -- start 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31aa1c1700 1 -- start start 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31aa1c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a4101080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31aa1c1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40fca70 con 0x7f31a40275d0 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a4101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a4101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44526/0 (socket says 172.21.15.38:44526) 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31a37fe700 1 -- 172.21.15.38:0/995847204 learned_addr learned my addr 172.21.15.38:0/995847204 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31a37fe700 1 -- 172.21.15.38:0/995847204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4101570 con 0x7f31a40275d0 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.773+0000 7f31a37fe700 1 --2- 172.21.15.38:0/995847204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a4101080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f319400b5f0 tx=0x7f3194006ee0).ready entity=mon.0 client_cookie=9b3cec1d0c71409 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31a27fc700 1 -- 172.21.15.38:0/995847204 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319400d600 con 0x7f31a40275d0 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31a27fc700 1 -- 172.21.15.38:0/995847204 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f319400dbc0 con 0x7f31a40275d0 2024-02-20T13:17:01.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31a27fc700 1 -- 172.21.15.38:0/995847204 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3194011b90 con 0x7f31a40275d0 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 -- 172.21.15.38:0/995847204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 msgr2=0x7f31a4101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 --2- 172.21.15.38:0/995847204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a4101080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f319400b5f0 tx=0x7f3194006ee0).stop 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 -- 172.21.15.38:0/995847204 shutdown_connections 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 --2- 172.21.15.38:0/995847204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a4101080 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 -- 172.21.15.38:0/995847204 >> 172.21.15.38:0/995847204 conn(0x7f31a40f6680 msgr2=0x7f31a40f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 -- 172.21.15.38:0/995847204 shutdown_connections 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.774+0000 7f31aa1c1700 1 -- 172.21.15.38:0/995847204 wait complete. 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31aa1c1700 1 Processor -- start 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31aa1c1700 1 -- start start 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31aa1c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31aa1c1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a410d400 con 0x7f31a40275d0 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a410cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44542/0 (socket says 172.21.15.38:44542) 2024-02-20T13:17:01.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.775+0000 7f31a37fe700 1 -- 172.21.15.38:0/1936624486 learned_addr learned my addr 172.21.15.38:0/1936624486 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f31a37fe700 1 -- 172.21.15.38:0/1936624486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319400b200 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f31a37fe700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a410cf10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f3194000c00 tx=0x7f3194006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319400de00 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a410d600 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f319401b870 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319401a670 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.776+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a410da90 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.777+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f319402b070 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.777+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4045bb0 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.777+0000 7f318bfff700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3190030fa0 0x7f3190033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.777+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3194042d00 con 0x7f31a40275d0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.777+0000 7f31a2ffd700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3190030fa0 0x7f3190033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:01.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.777+0000 7f31a2ffd700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3190030fa0 0x7f3190033460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f31980096a0 tx=0x7f3198006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:01.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.781+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f319402b6c0 con 0x7f31a40275d0 2024-02-20T13:17:01.947 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:01.946+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a410e3b0 con 0x7f3190030fa0 2024-02-20T13:17:02.390 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.388+0000 7f318bfff700 1 -- 172.21.15.38:0/1936624486 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31a410e3b0 con 0x7f3190030fa0 2024-02-20T13:17:02.392 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3190030fa0 msgr2=0x7f3190033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3190030fa0 0x7f3190033460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f31980096a0 tx=0x7f3198006b40).stop 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 msgr2=0x7f31a410cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a410cf10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f3194000c00 tx=0x7f3194006ee0).stop 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 shutdown_connections 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3190030fa0 0x7f3190033460 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 --2- 172.21.15.38:0/1936624486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31a40275d0 0x7f31a410cf10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 >> 172.21.15.38:0/1936624486 conn(0x7f31a40f6680 msgr2=0x7f31a40f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 shutdown_connections 2024-02-20T13:17:02.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:02.390+0000 7f31aa1c1700 1 -- 172.21.15.38:0/1936624486 wait complete. 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce326d700 1 Processor -- start 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce326d700 1 -- start start 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce326d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce326d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc072080 con 0x7f1cdc071c60 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce226b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce226b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44552/0 (socket says 172.21.15.38:44552) 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.632+0000 7f1ce226b700 1 -- 172.21.15.38:0/589675781 learned_addr learned my addr 172.21.15.38:0/589675781 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.633+0000 7f1ce226b700 1 -- 172.21.15.38:0/589675781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc070650 con 0x7f1cdc071c60 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.633+0000 7f1ce226b700 1 --2- 172.21.15.38:0/589675781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc070160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1cd800b5f0 tx=0x7f1cd8006ee0).ready entity=mon.0 client_cookie=51126d465d4003a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.633+0000 7f1ce1269700 1 -- 172.21.15.38:0/589675781 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd800d600 con 0x7f1cdc071c60 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.633+0000 7f1ce1269700 1 -- 172.21.15.38:0/589675781 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f1cd800dbc0 con 0x7f1cdc071c60 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.633+0000 7f1ce1269700 1 -- 172.21.15.38:0/589675781 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd8011b90 con 0x7f1cdc071c60 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 -- 172.21.15.38:0/589675781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 msgr2=0x7f1cdc070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 --2- 172.21.15.38:0/589675781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc070160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1cd800b5f0 tx=0x7f1cd8006ee0).stop 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 -- 172.21.15.38:0/589675781 shutdown_connections 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 --2- 172.21.15.38:0/589675781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc070160 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 -- 172.21.15.38:0/589675781 >> 172.21.15.38:0/589675781 conn(0x7f1cdc0f7600 msgr2=0x7f1cdc0f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 -- 172.21.15.38:0/589675781 shutdown_connections 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.634+0000 7f1ce326d700 1 -- 172.21.15.38:0/589675781 wait complete. 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce326d700 1 Processor -- start 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce326d700 1 -- start start 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce326d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce326d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc10d280 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce226b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce226b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc10cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44568/0 (socket says 172.21.15.38:44568) 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce226b700 1 -- 172.21.15.38:0/4159803892 learned_addr learned my addr 172.21.15.38:0/4159803892 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce226b700 1 -- 172.21.15.38:0/4159803892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd800b200 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.635+0000 7f1ce226b700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc10cd90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1cd800e040 tx=0x7f1cd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.636+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd801b570 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.636+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc10d480 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.636+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f1cd801bb30 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.636+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd801a820 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.636+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc10e0c0 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.636+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f1cd802b070 con 0x7f1cdc071c60 2024-02-20T13:17:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.637+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc10d910 con 0x7f1cdc071c60 2024-02-20T13:17:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.637+0000 7f1ccaffd700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f1cc4030fa0 0x7f1cc4033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.637+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f1cd8042df0 con 0x7f1cdc071c60 2024-02-20T13:17:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.637+0000 7f1ce1a6a700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f1cc4030fa0 0x7f1cc4033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.637+0000 7f1ce1a6a700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f1cc4030fa0 0x7f1cc4033460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1ccc0096a0 tx=0x7f1ccc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:03.642 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.641+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1cd802b680 con 0x7f1cdc071c60 2024-02-20T13:17:03.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:01.948045+0000 mgr.smithi038.powepi (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2024-02-20T13:17:03.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:02.382472+0000 mon.smithi038 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:03.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:02.388987+0000 mon.smithi038 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:03.805 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.804+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cdc102170 con 0x7f1cc4030fa0 2024-02-20T13:17:03.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.804+0000 7f1ccaffd700 1 -- 172.21.15.38:0/4159803892 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f1cdc102170 con 0x7f1cc4030fa0 2024-02-20T13:17:03.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3CKiYEjJ9Oe1YYj8RpxZWAbVJefJR5/FmzuhAFkhkPWUYDV1UlcRSTABPppqJkbOevKKNn27AIgSBZKp9vpyK7XBfnpvkWnBf6uvrmCNfBMASJgKCuUyEkzeMa0WWA1tv47FAQ3Kg9r1c7zPdc23pPErRlepbINL1SOWnXS/FoKe+7KeG0XpLqlQP4mZeNPFqp4LBP/xTOUYzhWClJ5AsQnzPD6ysQkbl6cDlaN/8o+ZopCTJpA5iB6rG9++RcclXYplZyTBflLN17LTebHVAV9XWyVLDhuRPH8BgKYUT0g7ZmJu+FumU67VsAHsG2bNhKP5X6f0EUN/4k5OUa/FpGTzQQZyeugpzgOye5gU9DHTOL33F0WcafCGRiUMRPawGFHiIdqm+Nvy/6v39Uo3Q1xiI33IYcXd51i1XOr+Lt+MDCDoNeENhToutWQJPPlqfdRdBTpXEfHEWb31POGJnXFTepEgG2P3CedbEb21758erC+fhiZu8BIyod7TlR/E= ceph-24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f1cc4030fa0 msgr2=0x7f1cc4033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f1cc4030fa0 0x7f1cc4033460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1ccc0096a0 tx=0x7f1ccc006b40).stop 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 msgr2=0x7f1cdc10cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc10cd90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1cd800e040 tx=0x7f1cd8006ee0).stop 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 shutdown_connections 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f1cc4030fa0 0x7f1cc4033460 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 --2- 172.21.15.38:0/4159803892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cdc071c60 0x7f1cdc10cd90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 >> 172.21.15.38:0/4159803892 conn(0x7f1cdc0f7600 msgr2=0x7f1cdc0f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 shutdown_connections 2024-02-20T13:17:03.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:03.806+0000 7f1ce326d700 1 -- 172.21.15.38:0/4159803892 wait complete. 2024-02-20T13:17:04.410 INFO:teuthology.orchestra.run.smithi038.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-20T13:17:04.410 INFO:teuthology.orchestra.run.smithi038.stderr:Adding key to root@localhost authorized_keys... 2024-02-20T13:17:04.410 INFO:teuthology.orchestra.run.smithi038.stderr:Adding host smithi038... 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f3066bb3700 1 Processor -- start 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f3066bb3700 1 -- start start 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f3066bb3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f30600fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f3066bb3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30600fb7e0 con 0x7f30600faed0 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f306494f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f30600fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f306494f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f30600fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44580/0 (socket says 172.21.15.38:44580) 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.981+0000 7f306494f700 1 -- 172.21.15.38:0/1304659005 learned_addr learned my addr 172.21.15.38:0/1304659005 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.982+0000 7f306494f700 1 -- 172.21.15.38:0/1304659005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30600fb920 con 0x7f30600faed0 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.982+0000 7f306494f700 1 --2- 172.21.15.38:0/1304659005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f30600fb2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f305400b870 tx=0x7f3054006f70).ready entity=mon.0 client_cookie=e54e10bc3b24c7b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.982+0000 7f305f7fe700 1 -- 172.21.15.38:0/1304659005 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f305400da20 con 0x7f30600faed0 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.982+0000 7f305f7fe700 1 -- 172.21.15.38:0/1304659005 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f3054010470 con 0x7f30600faed0 2024-02-20T13:17:04.987 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.982+0000 7f305f7fe700 1 -- 172.21.15.38:0/1304659005 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f305400f5b0 con 0x7f30600faed0 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 -- 172.21.15.38:0/1304659005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 msgr2=0x7f30600fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 --2- 172.21.15.38:0/1304659005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f30600fb2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f305400b870 tx=0x7f3054006f70).stop 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 -- 172.21.15.38:0/1304659005 shutdown_connections 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 --2- 172.21.15.38:0/1304659005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f30600fb2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 -- 172.21.15.38:0/1304659005 >> 172.21.15.38:0/1304659005 conn(0x7f30600f6680 msgr2=0x7f30600f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 -- 172.21.15.38:0/1304659005 shutdown_connections 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 -- 172.21.15.38:0/1304659005 wait complete. 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 Processor -- start 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.983+0000 7f3066bb3700 1 -- start start 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f3066bb3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f3060108b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f3066bb3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3060109040 con 0x7f30600faed0 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f306494f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f3060108b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f306494f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f3060108b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44594/0 (socket says 172.21.15.38:44594) 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f306494f700 1 -- 172.21.15.38:0/2682640785 learned_addr learned my addr 172.21.15.38:0/2682640785 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:04.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f306494f700 1 -- 172.21.15.38:0/2682640785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305400b480 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f306494f700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f3060108b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3054010d50 tx=0x7f3054006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f305400f3d0 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f3066bb3700 1 -- 172.21.15.38:0/2682640785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3060109240 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f3066bb3700 1 -- 172.21.15.38:0/2682640785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30601096d0 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f3054006c10 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.984+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3054019b60 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.985+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3038004fa0 con 0x7f30600faed0 2024-02-20T13:17:04.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.985+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f3054016030 con 0x7f30600faed0 2024-02-20T13:17:04.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.986+0000 7f305d7fa700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3044030fa0 0x7f3044033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:04.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.986+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3054045480 con 0x7f30600faed0 2024-02-20T13:17:04.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.986+0000 7f305ffff700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3044030fa0 0x7f3044033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:04.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.986+0000 7f305ffff700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3044030fa0 0x7f3044033460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f304c0096a0 tx=0x7f304c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:04.991 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:04.990+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3054021370 con 0x7f30600faed0 2024-02-20T13:17:05.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.154+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) v1 -- 0x7f3038000bc0 con 0x7f3044030fa0 2024-02-20T13:17:05.178 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:03.805690+0000 mgr.smithi038.powepi (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:05.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Added host 'smithi038' with addr '172.21.15.38' 2024-02-20T13:17:05.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.770+0000 7f305d7fa700 1 -- 172.21.15.38:0/2682640785 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f3038000bc0 con 0x7f3044030fa0 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.772+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3044030fa0 msgr2=0x7f3044033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.772+0000 7f304affd700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3044030fa0 0x7f3044033460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f304c0096a0 tx=0x7f304c006b40).stop 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.772+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 msgr2=0x7f3060108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.772+0000 7f304affd700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f3060108b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3054010d50 tx=0x7f3054006f70).stop 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.773+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 shutdown_connections 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.773+0000 7f304affd700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3044030fa0 0x7f3044033460 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.773+0000 7f304affd700 1 --2- 172.21.15.38:0/2682640785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30600faed0 0x7f3060108b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.773+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 >> 172.21.15.38:0/2682640785 conn(0x7f30600f6680 msgr2=0x7f30600ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.773+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 shutdown_connections 2024-02-20T13:17:05.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:05.773+0000 7f304affd700 1 -- 172.21.15.38:0/2682640785 wait complete. 2024-02-20T13:17:06.115 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:05.156563+0000 mgr.smithi038.powepi (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:06.400 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying mon service with default placement... 2024-02-20T13:17:06.919 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f803ad53700 1 Processor -- start 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f803ad53700 1 -- start start 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f803ad53700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f80340fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f803ad53700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340fc8d0 con 0x7f80340fbfc0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f8038aef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f80340fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f8038aef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f80340fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44610/0 (socket says 172.21.15.38:44610) 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f8038aef700 1 -- 172.21.15.38:0/556473665 learned_addr learned my addr 172.21.15.38:0/556473665 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.916+0000 7f8038aef700 1 -- 172.21.15.38:0/556473665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80340fca10 con 0x7f80340fbfc0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f8038aef700 1 --2- 172.21.15.38:0/556473665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f80340fc3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f802c009960 tx=0x7f802c0092d0).ready entity=mon.0 client_cookie=cb75f9b3aee46993 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f80337fe700 1 -- 172.21.15.38:0/556473665 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802c00b9c0 con 0x7f80340fbfc0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f80337fe700 1 -- 172.21.15.38:0/556473665 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f802c011400 con 0x7f80340fbfc0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f80337fe700 1 -- 172.21.15.38:0/556473665 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802c010580 con 0x7f80340fbfc0 2024-02-20T13:17:06.920 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f803ad53700 1 -- 172.21.15.38:0/556473665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 msgr2=0x7f80340fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f803ad53700 1 --2- 172.21.15.38:0/556473665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f80340fc3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f802c009960 tx=0x7f802c0092d0).stop 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f803ad53700 1 -- 172.21.15.38:0/556473665 shutdown_connections 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f803ad53700 1 --2- 172.21.15.38:0/556473665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f80340fc3e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.917+0000 7f803ad53700 1 -- 172.21.15.38:0/556473665 >> 172.21.15.38:0/556473665 conn(0x7f80340f7770 msgr2=0x7f80340f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.918+0000 7f803ad53700 1 -- 172.21.15.38:0/556473665 shutdown_connections 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.918+0000 7f803ad53700 1 -- 172.21.15.38:0/556473665 wait complete. 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.918+0000 7f803ad53700 1 Processor -- start 2024-02-20T13:17:06.921 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.918+0000 7f803ad53700 1 -- start start 2024-02-20T13:17:06.922 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f803ad53700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f803410ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:06.922 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f803ad53700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803410b230 con 0x7f80340fbfc0 2024-02-20T13:17:06.922 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f8038aef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f803410ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f8038aef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f803410ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44618/0 (socket says 172.21.15.38:44618) 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f8038aef700 1 -- 172.21.15.38:0/577030660 learned_addr learned my addr 172.21.15.38:0/577030660 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f8038aef700 1 -- 172.21.15.38:0/577030660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c009570 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f8038aef700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f803410ad40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f802c009fd0 tx=0x7f802c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802c010840 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f802c010e00 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803410b430 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802c01bdf0 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.919+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803410c040 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.920+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f802c00f070 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.920+0000 7f80317fa700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8014030fa0 0x7f8014033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.920+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f802c042fe0 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.920+0000 7f8033fff700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8014030fa0 0x7f8014033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.921+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80340591e0 con 0x7f80340fbfc0 2024-02-20T13:17:06.923 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.921+0000 7f8033fff700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8014030fa0 0x7f8014033460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f801c0096a0 tx=0x7f801c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:06.925+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f802c011e60 con 0x7f80340fbfc0 2024-02-20T13:17:07.095 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.094+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f80340fde10 con 0x7f8014030fa0 2024-02-20T13:17:07.104 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.103+0000 7f80317fa700 1 -- 172.21.15.38:0/577030660 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f80340fde10 con 0x7f8014030fa0 2024-02-20T13:17:07.104 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-20T13:17:07.106 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8014030fa0 msgr2=0x7f8014033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8014030fa0 0x7f8014033460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f801c0096a0 tx=0x7f801c006b40).stop 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 msgr2=0x7f803410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f803410ad40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f802c009fd0 tx=0x7f802c0092d0).stop 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 shutdown_connections 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f8014030fa0 0x7f8014033460 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 --2- 172.21.15.38:0/577030660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80340fbfc0 0x7f803410ad40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 >> 172.21.15.38:0/577030660 conn(0x7f80340f7770 msgr2=0x7f80340f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.104+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 shutdown_connections 2024-02-20T13:17:07.107 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:07.105+0000 7f803ad53700 1 -- 172.21.15.38:0/577030660 wait complete. 2024-02-20T13:17:07.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:06 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:05.771032+0000 mon.smithi038 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:07.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:06 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:05.771350+0000 mgr.smithi038.powepi (mgr.14116) 9 : cephadm [INF] Added host smithi038 2024-02-20T13:17:07.125 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:06 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:05.771605+0000 mon.smithi038 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:17:07.672 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying mgr service with default placement... 2024-02-20T13:17:08.195 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:07.095948+0000 mgr.smithi038.powepi (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:08.195 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:07.096686+0000 mgr.smithi038.powepi (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-20T13:17:08.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:07.104006+0000 mon.smithi038 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.192+0000 7f05d93c5700 1 Processor -- start 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.192+0000 7f05d93c5700 1 -- start start 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.192+0000 7f05d93c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.192+0000 7f05d93c5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40ff0d0 con 0x7f05d4027670 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.192+0000 7f05d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58316/0 (socket says 172.21.15.38:58316) 2024-02-20T13:17:08.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d2ffd700 1 -- 172.21.15.38:0/2732518078 learned_addr learned my addr 172.21.15.38:0/2732518078 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d2ffd700 1 -- 172.21.15.38:0/2732518078 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40ff210 con 0x7f05d4027670 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d2ffd700 1 --2- 172.21.15.38:0/2732518078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40febe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f05c800b5f0 tx=0x7f05c8006ee0).ready entity=mon.0 client_cookie=efdab235be6cac21 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d1ffb700 1 -- 172.21.15.38:0/2732518078 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c800d600 con 0x7f05d4027670 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d1ffb700 1 -- 172.21.15.38:0/2732518078 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f05c800dbc0 con 0x7f05d4027670 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.193+0000 7f05d1ffb700 1 -- 172.21.15.38:0/2732518078 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c8011b90 con 0x7f05d4027670 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 -- 172.21.15.38:0/2732518078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 msgr2=0x7f05d40febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 --2- 172.21.15.38:0/2732518078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40febe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f05c800b5f0 tx=0x7f05c8006ee0).stop 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 -- 172.21.15.38:0/2732518078 shutdown_connections 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 --2- 172.21.15.38:0/2732518078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40febe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 -- 172.21.15.38:0/2732518078 >> 172.21.15.38:0/2732518078 conn(0x7f05d40f6680 msgr2=0x7f05d40f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 -- 172.21.15.38:0/2732518078 shutdown_connections 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 -- 172.21.15.38:0/2732518078 wait complete. 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 Processor -- start 2024-02-20T13:17:08.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.194+0000 7f05d93c5700 1 -- start start 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d93c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d93c5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40faae0 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58318/0 (socket says 172.21.15.38:58318) 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d2ffd700 1 -- 172.21.15.38:0/327423454 learned_addr learned my addr 172.21.15.38:0/327423454 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d2ffd700 1 -- 172.21.15.38:0/327423454 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c800b200 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d2ffd700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40fc660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f05c8006800 tx=0x7f05c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c800de50 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f05c801b870 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d40fac80 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c801a7a0 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.195+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d40fb110 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.196+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f05c800f070 con 0x7f05d4027670 2024-02-20T13:17:08.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.196+0000 7f05bb7fe700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f05c0030fa0 0x7f05c0033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:08.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.196+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d4045bb0 con 0x7f05d4027670 2024-02-20T13:17:08.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.196+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f05c8042bf0 con 0x7f05d4027670 2024-02-20T13:17:08.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.196+0000 7f05d27fc700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f05c0030fa0 0x7f05c0033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:08.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.197+0000 7f05d27fc700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f05c0030fa0 0x7f05c0033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f05c40096a0 tx=0x7f05c4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:08.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.201+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f05c8016030 con 0x7f05d4027670 2024-02-20T13:17:08.371 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.370+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f05d40fb350 con 0x7f05c0030fa0 2024-02-20T13:17:08.380 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.379+0000 7f05bb7fe700 1 -- 172.21.15.38:0/327423454 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f05d40fb350 con 0x7f05c0030fa0 2024-02-20T13:17:08.380 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f05c0030fa0 msgr2=0x7f05c0033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f05c0030fa0 0x7f05c0033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f05c40096a0 tx=0x7f05c4006b40).stop 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 msgr2=0x7f05d40fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40fc660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f05c8006800 tx=0x7f05c8006ee0).stop 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 shutdown_connections 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f05c0030fa0 0x7f05c0033460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 --2- 172.21.15.38:0/327423454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d4027670 0x7f05d40fc660 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 >> 172.21.15.38:0/327423454 conn(0x7f05d40f6680 msgr2=0x7f05d40f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 shutdown_connections 2024-02-20T13:17:08.383 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:08.381+0000 7f05d93c5700 1 -- 172.21.15.38:0/327423454 wait complete. 2024-02-20T13:17:08.982 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying crash service with default placement... 2024-02-20T13:17:09.531 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.526+0000 7f561d85d700 1 Processor -- start 2024-02-20T13:17:09.531 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.526+0000 7f561d85d700 1 -- start start 2024-02-20T13:17:09.531 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f561d85d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f56180fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:09.531 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f561d85d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56180fc8d0 con 0x7f56180fbfc0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5616ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f56180fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5616ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f56180fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58322/0 (socket says 172.21.15.38:58322) 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5616ffd700 1 -- 172.21.15.38:0/367681981 learned_addr learned my addr 172.21.15.38:0/367681981 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5616ffd700 1 -- 172.21.15.38:0/367681981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56180fca10 con 0x7f56180fbfc0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5616ffd700 1 --2- 172.21.15.38:0/367681981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f56180fc3e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f560800b5f0 tx=0x7f5608006ee0).ready entity=mon.0 client_cookie=4ba861278864a01f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5615ffb700 1 -- 172.21.15.38:0/367681981 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560800d600 con 0x7f56180fbfc0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.527+0000 7f5615ffb700 1 -- 172.21.15.38:0/367681981 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f560800dbc0 con 0x7f56180fbfc0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f5615ffb700 1 -- 172.21.15.38:0/367681981 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5608011b90 con 0x7f56180fbfc0 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 -- 172.21.15.38:0/367681981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 msgr2=0x7f56180fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 --2- 172.21.15.38:0/367681981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f56180fc3e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f560800b5f0 tx=0x7f5608006ee0).stop 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 -- 172.21.15.38:0/367681981 shutdown_connections 2024-02-20T13:17:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 --2- 172.21.15.38:0/367681981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f56180fc3e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 -- 172.21.15.38:0/367681981 >> 172.21.15.38:0/367681981 conn(0x7f56180f7770 msgr2=0x7f56180f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 -- 172.21.15.38:0/367681981 shutdown_connections 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.528+0000 7f561d85d700 1 -- 172.21.15.38:0/367681981 wait complete. 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.529+0000 7f561d85d700 1 Processor -- start 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.529+0000 7f561d85d700 1 -- start start 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.529+0000 7f561d85d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f561810ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.529+0000 7f561d85d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561810b280 con 0x7f56180fbfc0 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f5616ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f561810ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f5616ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f561810ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58330/0 (socket says 172.21.15.38:58330) 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f5616ffd700 1 -- 172.21.15.38:0/4071508517 learned_addr learned my addr 172.21.15.38:0/4071508517 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f5616ffd700 1 -- 172.21.15.38:0/4071508517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560800b200 con 0x7f56180fbfc0 2024-02-20T13:17:09.534 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f5616ffd700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f561810ad90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5608006800 tx=0x7f5608006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560800de50 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f560801b870 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561810b480 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560801a7a0 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.530+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561810c090 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.531+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f560800f070 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.531+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56180591e0 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.531+0000 7f560f7fe700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f55f8030fa0 0x7f55f8033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.531+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f5608042bf0 con 0x7f56180fbfc0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.531+0000 7f56167fc700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f55f8030fa0 0x7f55f8033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.532+0000 7f56167fc700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f55f8030fa0 0x7f55f8033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f56000096a0 tx=0x7f5600006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:09.537 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.536+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f560802b6b0 con 0x7f56180fbfc0 2024-02-20T13:17:09.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:08.372268+0000 mgr.smithi038.powepi (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:08.372786+0000 mgr.smithi038.powepi (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-20T13:17:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:08.380113+0000 mon.smithi038 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:09.707 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.705+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f561810c2d0 con 0x7f55f8030fa0 2024-02-20T13:17:09.716 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.714+0000 7f560f7fe700 1 -- 172.21.15.38:0/4071508517 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f561810c2d0 con 0x7f55f8030fa0 2024-02-20T13:17:09.716 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f55f8030fa0 msgr2=0x7f55f8033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f55f8030fa0 0x7f55f8033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f56000096a0 tx=0x7f5600006b40).stop 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 msgr2=0x7f561810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f561810ad90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5608006800 tx=0x7f5608006ee0).stop 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 shutdown_connections 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f55f8030fa0 0x7f55f8033460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:09.718 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 --2- 172.21.15.38:0/4071508517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56180fbfc0 0x7f561810ad90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:09.719 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 >> 172.21.15.38:0/4071508517 conn(0x7f56180f7770 msgr2=0x7f56180f9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:09.719 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 shutdown_connections 2024-02-20T13:17:09.719 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:09.716+0000 7f561d85d700 1 -- 172.21.15.38:0/4071508517 wait complete. 2024-02-20T13:17:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying prometheus service with default placement... 2024-02-20T13:17:11.167 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:09.707881+0000 mgr.smithi038.powepi (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:11.168 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:09.708541+0000 mgr.smithi038.powepi (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-20T13:17:11.168 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:09.715453+0000 mon.smithi038 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.463+0000 7fc89c49f700 1 Processor -- start 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.463+0000 7fc89c49f700 1 -- start start 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89c49f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc8940fa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89c49f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8940fabf0 con 0x7fc8940fa2e0 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89a23b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc8940fa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89a23b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc8940fa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58340/0 (socket says 172.21.15.38:58340) 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89a23b700 1 -- 172.21.15.38:0/539238367 learned_addr learned my addr 172.21.15.38:0/539238367 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89a23b700 1 -- 172.21.15.38:0/539238367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8940fad30 con 0x7fc8940fa2e0 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc89a23b700 1 --2- 172.21.15.38:0/539238367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc8940fa700 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc88400b5f0 tx=0x7fc884006ee0).ready entity=mon.0 client_cookie=b9f7ac19feabd976 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc899239700 1 -- 172.21.15.38:0/539238367 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc88400d600 con 0x7fc8940fa2e0 2024-02-20T13:17:11.467 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc899239700 1 -- 172.21.15.38:0/539238367 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc88400dbc0 con 0x7fc8940fa2e0 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.464+0000 7fc899239700 1 -- 172.21.15.38:0/539238367 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc884011b90 con 0x7fc8940fa2e0 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 -- 172.21.15.38:0/539238367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 msgr2=0x7fc8940fa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 --2- 172.21.15.38:0/539238367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc8940fa700 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc88400b5f0 tx=0x7fc884006ee0).stop 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 -- 172.21.15.38:0/539238367 shutdown_connections 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 --2- 172.21.15.38:0/539238367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc8940fa700 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 -- 172.21.15.38:0/539238367 >> 172.21.15.38:0/539238367 conn(0x7fc8940f5e20 msgr2=0x7fc8940f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 -- 172.21.15.38:0/539238367 shutdown_connections 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.465+0000 7fc89c49f700 1 -- 172.21.15.38:0/539238367 wait complete. 2024-02-20T13:17:11.468 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89c49f700 1 Processor -- start 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89c49f700 1 -- start start 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89c49f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc89410cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89c49f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89410d3e0 con 0x7fc8940fa2e0 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89a23b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc89410cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89a23b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc89410cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58356/0 (socket says 172.21.15.38:58356) 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89a23b700 1 -- 172.21.15.38:0/3132070676 learned_addr learned my addr 172.21.15.38:0/3132070676 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89a23b700 1 -- 172.21.15.38:0/3132070676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88400b200 con 0x7fc8940fa2e0 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.466+0000 7fc89a23b700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc89410cef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fc884000c00 tx=0x7fc884006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.467+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc88401b410 con 0x7fc8940fa2e0 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.467+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc88401b9d0 con 0x7fc8940fa2e0 2024-02-20T13:17:11.470 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.467+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89410d5e0 con 0x7fc8940fa2e0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.467+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc88401a600 con 0x7fc8940fa2e0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.467+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89410da70 con 0x7fc8940fa2e0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.467+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fc88402b070 con 0x7fc8940fa2e0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.468+0000 7fc882ffd700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fc87c030fa0 0x7fc87c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.468+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fc884042c20 con 0x7fc8940fa2e0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.468+0000 7fc899a3a700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fc87c030fa0 0x7fc87c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.468+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc894045bb0 con 0x7fc8940fa2e0 2024-02-20T13:17:11.471 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.468+0000 7fc899a3a700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fc87c030fa0 0x7fc87c033460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc8900096a0 tx=0x7fc890006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:11.474 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.472+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc88401f070 con 0x7fc8940fa2e0 2024-02-20T13:17:11.641 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.640+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8940fb950 con 0x7fc87c030fa0 2024-02-20T13:17:11.650 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.648+0000 7fc882ffd700 1 -- 172.21.15.38:0/3132070676 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc8940fb950 con 0x7fc87c030fa0 2024-02-20T13:17:11.650 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-20T13:17:11.652 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fc87c030fa0 msgr2=0x7fc87c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:11.652 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fc87c030fa0 0x7fc87c033460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc8900096a0 tx=0x7fc890006b40).stop 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 msgr2=0x7fc89410cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc89410cef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fc884000c00 tx=0x7fc884006ee0).stop 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 shutdown_connections 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7fc87c030fa0 0x7fc87c033460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 --2- 172.21.15.38:0/3132070676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8940fa2e0 0x7fc89410cef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.650+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 >> 172.21.15.38:0/3132070676 conn(0x7fc8940f5e20 msgr2=0x7fc8940fb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.651+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 shutdown_connections 2024-02-20T13:17:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:11.651+0000 7fc89c49f700 1 -- 172.21.15.38:0/3132070676 wait complete. 2024-02-20T13:17:12.307 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying grafana service with default placement... 2024-02-20T13:17:12.945 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:11.641959+0000 mgr.smithi038.powepi (mgr.14116) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:12.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:11.642481+0000 mgr.smithi038.powepi (mgr.14116) 17 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-20T13:17:12.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:11.649719+0000 mon.smithi038 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.927+0000 7f767abe3700 1 Processor -- start 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.927+0000 7f767abe3700 1 -- start start 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.927+0000 7f767abe3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f7674071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.927+0000 7f767abe3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76740730b0 con 0x7f7674072c90 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f767897f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f7674071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f767897f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f7674071230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58358/0 (socket says 172.21.15.38:58358) 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f767897f700 1 -- 172.21.15.38:0/4173391612 learned_addr learned my addr 172.21.15.38:0/4173391612 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f767897f700 1 -- 172.21.15.38:0/4173391612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674071720 con 0x7f7674072c90 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f767897f700 1 --2- 172.21.15.38:0/4173391612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f7674071230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f766c009960 tx=0x7f766c0092d0).ready entity=mon.0 client_cookie=bc42db0421696b1a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:13.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f76737fe700 1 -- 172.21.15.38:0/4173391612 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f766c00b9c0 con 0x7f7674072c90 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f76737fe700 1 -- 172.21.15.38:0/4173391612 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f766c011400 con 0x7f7674072c90 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.928+0000 7f76737fe700 1 -- 172.21.15.38:0/4173391612 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f766c010580 con 0x7f7674072c90 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 -- 172.21.15.38:0/4173391612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 msgr2=0x7f7674071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 --2- 172.21.15.38:0/4173391612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f7674071230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f766c009960 tx=0x7f766c0092d0).stop 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 -- 172.21.15.38:0/4173391612 shutdown_connections 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 --2- 172.21.15.38:0/4173391612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f7674071230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 -- 172.21.15.38:0/4173391612 >> 172.21.15.38:0/4173391612 conn(0x7f76740f7770 msgr2=0x7f76740f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 -- 172.21.15.38:0/4173391612 shutdown_connections 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 -- 172.21.15.38:0/4173391612 wait complete. 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 Processor -- start 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.929+0000 7f767abe3700 1 -- start start 2024-02-20T13:17:13.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767abe3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f767410cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767abe3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767410d480 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767897f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f767410cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767897f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f767410cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58360/0 (socket says 172.21.15.38:58360) 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767897f700 1 -- 172.21.15.38:0/734593702 learned_addr learned my addr 172.21.15.38:0/734593702 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767897f700 1 -- 172.21.15.38:0/734593702 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c009570 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f767897f700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f767410cf90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f766c011c40 tx=0x7f766c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f766c010840 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.930+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f766c010e00 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.931+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767410d680 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.931+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f766c01acb0 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.931+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767410db10 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.931+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f766c02a070 con 0x7f7674072c90 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.931+0000 7f76717fa700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f7654030fa0 0x7f7654033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:13.017 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.931+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7674045bb0 con 0x7f7674072c90 2024-02-20T13:17:13.018 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.932+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f766c042ff0 con 0x7f7674072c90 2024-02-20T13:17:13.018 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.932+0000 7f7673fff700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f7654030fa0 0x7f7654033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:13.018 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.932+0000 7f7673fff700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f7654030fa0 0x7f7654033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f765c0096a0 tx=0x7f765c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:13.018 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:12.936+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f766c01a530 con 0x7f7674072c90 2024-02-20T13:17:13.118 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.117+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f767410e370 con 0x7f7654030fa0 2024-02-20T13:17:13.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.122+0000 7f76717fa700 1 -- 172.21.15.38:0/734593702 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f767410e370 con 0x7f7654030fa0 2024-02-20T13:17:13.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f7654030fa0 msgr2=0x7f7654033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f7654030fa0 0x7f7654033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f765c0096a0 tx=0x7f765c006b40).stop 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 msgr2=0x7f767410cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f767410cf90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f766c011c40 tx=0x7f766c0092d0).stop 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 shutdown_connections 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f7654030fa0 0x7f7654033460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 --2- 172.21.15.38:0/734593702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7674072c90 0x7f767410cf90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.124+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 >> 172.21.15.38:0/734593702 conn(0x7f76740f7770 msgr2=0x7f76740f92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.125+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 shutdown_connections 2024-02-20T13:17:13.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:13.125+0000 7f767abe3700 1 -- 172.21.15.38:0/734593702 wait complete. 2024-02-20T13:17:13.826 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying node-exporter service with default placement... 2024-02-20T13:17:14.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:13.119336+0000 mgr.smithi038.powepi (mgr.14116) 18 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:14.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:13.119861+0000 mgr.smithi038.powepi (mgr.14116) 19 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-20T13:17:14.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:13.123913+0000 mon.smithi038 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:14.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.588+0000 7f68939e3700 1 Processor -- start 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f68939e3700 1 -- start start 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f68939e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f68939e3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0ff0d0 con 0x7f688c027670 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f689177f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f689177f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58372/0 (socket says 172.21.15.38:58372) 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f689177f700 1 -- 172.21.15.38:0/2862543752 learned_addr learned my addr 172.21.15.38:0/2862543752 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f689177f700 1 -- 172.21.15.38:0/2862543752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0ff210 con 0x7f688c027670 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f689177f700 1 --2- 172.21.15.38:0/2862543752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0febe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f688800b5f0 tx=0x7f6888006ee0).ready entity=mon.0 client_cookie=946aaea1fe35c49c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f687ffff700 1 -- 172.21.15.38:0/2862543752 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f688800d600 con 0x7f688c027670 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f687ffff700 1 -- 172.21.15.38:0/2862543752 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f688800dbc0 con 0x7f688c027670 2024-02-20T13:17:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.589+0000 7f687ffff700 1 -- 172.21.15.38:0/2862543752 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6888011b90 con 0x7f688c027670 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 -- 172.21.15.38:0/2862543752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 msgr2=0x7f688c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 --2- 172.21.15.38:0/2862543752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0febe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f688800b5f0 tx=0x7f6888006ee0).stop 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 -- 172.21.15.38:0/2862543752 shutdown_connections 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 --2- 172.21.15.38:0/2862543752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0febe0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 -- 172.21.15.38:0/2862543752 >> 172.21.15.38:0/2862543752 conn(0x7f688c0f6680 msgr2=0x7f688c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 -- 172.21.15.38:0/2862543752 shutdown_connections 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.590+0000 7f68939e3700 1 -- 172.21.15.38:0/2862543752 wait complete. 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f68939e3700 1 Processor -- start 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f68939e3700 1 -- start start 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f68939e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f68939e3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0faae0 con 0x7f688c027670 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f689177f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f689177f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58380/0 (socket says 172.21.15.38:58380) 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f689177f700 1 -- 172.21.15.38:0/1574099275 learned_addr learned my addr 172.21.15.38:0/1574099275 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f689177f700 1 -- 172.21.15.38:0/1574099275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688800b200 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f689177f700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0fc660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6888000c00 tx=0x7f6888006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f688801b410 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f688801b9d0 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c0fac80 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.591+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f688801a620 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.592+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c0fb110 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.593+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f688802b070 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.593+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c045bb0 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.593+0000 7f687dffb700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f6874030fa0 0x7f6874033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.593+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6888042c20 con 0x7f688c027670 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.593+0000 7f6890f7e700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f6874030fa0 0x7f6874033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.593+0000 7f6890f7e700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f6874030fa0 0x7f6874033460 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f68800096a0 tx=0x7f6880006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.597+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f688802b950 con 0x7f688c027670 2024-02-20T13:17:14.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.771+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f688c100090 con 0x7f6874030fa0 2024-02-20T13:17:14.778 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.777+0000 7f687dffb700 1 -- 172.21.15.38:0/1574099275 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f688c100090 con 0x7f6874030fa0 2024-02-20T13:17:14.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-20T13:17:14.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f6874030fa0 msgr2=0x7f6874033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:14.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f6874030fa0 0x7f6874033460 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f68800096a0 tx=0x7f6880006b40).stop 2024-02-20T13:17:14.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 msgr2=0x7f688c0fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:14.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0fc660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6888000c00 tx=0x7f6888006ee0).stop 2024-02-20T13:17:14.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 shutdown_connections 2024-02-20T13:17:14.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f6874030fa0 0x7f6874033460 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:14.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 --2- 172.21.15.38:0/1574099275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f688c027670 0x7f688c0fc660 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:14.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 >> 172.21.15.38:0/1574099275 conn(0x7f688c0f6680 msgr2=0x7f688c0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:14.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 shutdown_connections 2024-02-20T13:17:14.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:14.779+0000 7f68939e3700 1 -- 172.21.15.38:0/1574099275 wait complete. 2024-02-20T13:17:15.554 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying alertmanager service with default placement... 2024-02-20T13:17:15.820 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:14.773793+0000 mgr.smithi038.powepi (mgr.14116) 20 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:15.820 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:14.774353+0000 mgr.smithi038.powepi (mgr.14116) 21 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-20T13:17:15.821 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:14.778457+0000 mon.smithi038 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:15.821 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:15.032065+0000 mon.smithi038 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:16.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.082+0000 7f34a77a1700 1 Processor -- start 2024-02-20T13:17:16.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.082+0000 7f34a77a1700 1 -- start start 2024-02-20T13:17:16.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.082+0000 7f34a77a1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a00febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:16.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.082+0000 7f34a77a1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00ff0d0 con 0x7f34a0027670 2024-02-20T13:17:16.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f34a553d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a00febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:16.086 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f34a553d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a00febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58384/0 (socket says 172.21.15.38:58384) 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f34a553d700 1 -- 172.21.15.38:0/1808035909 learned_addr learned my addr 172.21.15.38:0/1808035909 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f34a553d700 1 -- 172.21.15.38:0/1808035909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00ff210 con 0x7f34a0027670 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f34a553d700 1 --2- 172.21.15.38:0/1808035909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a00febe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f349c00b5f0 tx=0x7f349c006ee0).ready entity=mon.0 client_cookie=46e9d3dfa703ec71 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f348ffff700 1 -- 172.21.15.38:0/1808035909 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349c00d600 con 0x7f34a0027670 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f348ffff700 1 -- 172.21.15.38:0/1808035909 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f349c00dbf0 con 0x7f34a0027670 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.083+0000 7f348ffff700 1 -- 172.21.15.38:0/1808035909 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349c011bf0 con 0x7f34a0027670 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.084+0000 7f34a77a1700 1 -- 172.21.15.38:0/1808035909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 msgr2=0x7f34a00febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.084+0000 7f34a77a1700 1 --2- 172.21.15.38:0/1808035909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a00febe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f349c00b5f0 tx=0x7f349c006ee0).stop 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.084+0000 7f34a77a1700 1 -- 172.21.15.38:0/1808035909 shutdown_connections 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.084+0000 7f34a77a1700 1 --2- 172.21.15.38:0/1808035909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a00febe0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.084+0000 7f34a77a1700 1 -- 172.21.15.38:0/1808035909 >> 172.21.15.38:0/1808035909 conn(0x7f34a00f6680 msgr2=0x7f34a00f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.085+0000 7f34a77a1700 1 -- 172.21.15.38:0/1808035909 shutdown_connections 2024-02-20T13:17:16.087 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.085+0000 7f34a77a1700 1 -- 172.21.15.38:0/1808035909 wait complete. 2024-02-20T13:17:16.088 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.085+0000 7f34a77a1700 1 Processor -- start 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.085+0000 7f34a77a1700 1 -- start start 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.085+0000 7f34a77a1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a010ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.085+0000 7f34a77a1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a010b250 con 0x7f34a0027670 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f34a553d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a010ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f34a553d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a010ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58396/0 (socket says 172.21.15.38:58396) 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f34a553d700 1 -- 172.21.15.38:0/2252547430 learned_addr learned my addr 172.21.15.38:0/2252547430 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f34a553d700 1 -- 172.21.15.38:0/2252547430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c00b200 con 0x7f34a0027670 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f34a553d700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a010ad60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f349c011de0 tx=0x7f349c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349c01b410 con 0x7f34a0027670 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a010b450 con 0x7f34a0027670 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f349c01ba00 con 0x7f34a0027670 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.086+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349c01a770 con 0x7f34a0027670 2024-02-20T13:17:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.087+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a010c060 con 0x7f34a0027670 2024-02-20T13:17:16.090 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.087+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f349c02b070 con 0x7f34a0027670 2024-02-20T13:17:16.090 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.087+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a00591e0 con 0x7f34a0027670 2024-02-20T13:17:16.090 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.088+0000 7f348dffb700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3488030fa0 0x7f3488033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:16.090 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.088+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f349c042c30 con 0x7f34a0027670 2024-02-20T13:17:16.090 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.089+0000 7f34a4d3c700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3488030fa0 0x7f3488033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:16.091 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.089+0000 7f34a4d3c700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3488030fa0 0x7f3488033460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f34900096a0 tx=0x7f3490006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:16.093 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.092+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f349c02b8f0 con 0x7f34a0027670 2024-02-20T13:17:16.288 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.286+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a010c360 con 0x7f3488030fa0 2024-02-20T13:17:16.294 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.292+0000 7f348dffb700 1 -- 172.21.15.38:0/2252547430 <== mgr.14116 v2:172.21.15.38:6800/4148388115 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f34a010c360 con 0x7f3488030fa0 2024-02-20T13:17:16.294 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-20T13:17:16.296 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3488030fa0 msgr2=0x7f3488033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:16.296 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3488030fa0 0x7f3488033460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f34900096a0 tx=0x7f3490006b40).stop 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 msgr2=0x7f34a010ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a010ad60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f349c011de0 tx=0x7f349c006ee0).stop 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 shutdown_connections 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f3488030fa0 0x7f3488033460 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 --2- 172.21.15.38:0/2252547430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34a0027670 0x7f34a010ad60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 >> 172.21.15.38:0/2252547430 conn(0x7f34a00f6680 msgr2=0x7f34a00f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 shutdown_connections 2024-02-20T13:17:16.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:16.294+0000 7f34a77a1700 1 -- 172.21.15.38:0/2252547430 wait complete. 2024-02-20T13:17:17.195 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:15.888923+0000 mon.smithi038 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:17.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:16.288730+0000 mgr.smithi038.powepi (mgr.14116) 22 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:17.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:16.289306+0000 mgr.smithi038.powepi (mgr.14116) 23 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-20T13:17:17.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:16.293459+0000 mon.smithi038 (mon.0) 65 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:17.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:16.332860+0000 mon.smithi038 (mon.0) 66 : audit [INF] from='mgr.14116 172.21.15.38:0/1287030357' entity='mgr.smithi038.powepi' 2024-02-20T13:17:18.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:17.057793+0000 mgr.smithi038.powepi (mgr.14116) 24 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d14057700 1 Processor -- start 2024-02-20T13:17:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d14057700 1 -- start start 2024-02-20T13:17:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d14057700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d14057700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0ff130 con 0x7f9d0c027610 2024-02-20T13:17:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d11df3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d11df3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45734/0 (socket says 172.21.15.38:45734) 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.125+0000 7f9d11df3700 1 -- 172.21.15.38:0/4166431775 learned_addr learned my addr 172.21.15.38:0/4166431775 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d11df3700 1 -- 172.21.15.38:0/4166431775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c0ff270 con 0x7f9d0c027610 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d11df3700 1 --2- 172.21.15.38:0/4166431775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fec40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9d0800b5f0 tx=0x7f9d08006ee0).ready entity=mon.0 client_cookie=5b39cc7b7259b25f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d10df1700 1 -- 172.21.15.38:0/4166431775 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0800d600 con 0x7f9d0c027610 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d10df1700 1 -- 172.21.15.38:0/4166431775 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9d0800dbf0 con 0x7f9d0c027610 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d10df1700 1 -- 172.21.15.38:0/4166431775 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d08011bf0 con 0x7f9d0c027610 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d14057700 1 -- 172.21.15.38:0/4166431775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 msgr2=0x7f9d0c0fec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d14057700 1 --2- 172.21.15.38:0/4166431775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fec40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9d0800b5f0 tx=0x7f9d08006ee0).stop 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d14057700 1 -- 172.21.15.38:0/4166431775 shutdown_connections 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d14057700 1 --2- 172.21.15.38:0/4166431775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fec40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.126+0000 7f9d14057700 1 -- 172.21.15.38:0/4166431775 >> 172.21.15.38:0/4166431775 conn(0x7f9d0c0f6680 msgr2=0x7f9d0c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d14057700 1 -- 172.21.15.38:0/4166431775 shutdown_connections 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d14057700 1 -- 172.21.15.38:0/4166431775 wait complete. 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d14057700 1 Processor -- start 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d14057700 1 -- start start 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d14057700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d14057700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0fab40 con 0x7f9d0c027610 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d11df3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d11df3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45746/0 (socket says 172.21.15.38:45746) 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.127+0000 7f9d11df3700 1 -- 172.21.15.38:0/994847425 learned_addr learned my addr 172.21.15.38:0/994847425 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9d11df3700 1 -- 172.21.15.38:0/994847425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0800b200 con 0x7f9d0c027610 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9d11df3700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fc6c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9d08011de0 tx=0x7f9d08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0801b410 con 0x7f9d0c027610 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9d0801ba00 con 0x7f9d0c027610 2024-02-20T13:17:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c0face0 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0801a770 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.128+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c0fb170 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.129+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f9d0802b070 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.129+0000 7f9cfa7fc700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f9cf4030ff0 0x7f9cf40334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.129+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c045bb0 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.129+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f9d08042bc0 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.129+0000 7f9d115f2700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f9cf4030ff0 0x7f9cf40334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.129+0000 7f9d115f2700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f9cf4030ff0 0x7f9cf40334b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.133+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9d08016020 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.292+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f9d0c0ffee0 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.295+0000 7f9cfa7fc700 1 -- 172.21.15.38:0/994847425 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9d0801a9e0 con 0x7f9d0c027610 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f9cf4030ff0 msgr2=0x7f9cf40334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f9cf4030ff0 0x7f9cf40334b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).stop 2024-02-20T13:17:18.461 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 msgr2=0x7f9d0c0fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fc6c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9d08011de0 tx=0x7f9d08006ee0).stop 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 shutdown_connections 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f9cf4030ff0 0x7f9cf40334b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 --2- 172.21.15.38:0/994847425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d0c027610 0x7f9d0c0fc6c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 >> 172.21.15.38:0/994847425 conn(0x7f9d0c0f6680 msgr2=0x7f9d0c0f81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 shutdown_connections 2024-02-20T13:17:18.462 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:18.297+0000 7f9d14057700 1 -- 172.21.15.38:0/994847425 wait complete. 2024-02-20T13:17:19.325 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:18.296899+0000 mon.smithi038 (mon.0) 67 : audit [INF] from='client.? 172.21.15.38:0/994847425' entity='client.admin' 2024-02-20T13:17:19.624 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.621+0000 7f0fb3bf2700 1 Processor -- start 2024-02-20T13:17:19.624 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.621+0000 7f0fb3bf2700 1 -- start start 2024-02-20T13:17:19.624 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb3bf2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac101040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:19.624 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb3bf2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac0fca30 con 0x7f0fac06ad80 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb198e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb198e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45760/0 (socket says 172.21.15.38:45760) 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb198e700 1 -- 172.21.15.38:0/3398747192 learned_addr learned my addr 172.21.15.38:0/3398747192 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb198e700 1 -- 172.21.15.38:0/3398747192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac101530 con 0x7f0fac06ad80 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb198e700 1 --2- 172.21.15.38:0/3398747192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac101040 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0fa800b5f0 tx=0x7f0fa8006ee0).ready entity=mon.0 client_cookie=2045d05b8dcdb15a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb098c700 1 -- 172.21.15.38:0/3398747192 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa800d600 con 0x7f0fac06ad80 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb098c700 1 -- 172.21.15.38:0/3398747192 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0fa800dbf0 con 0x7f0fac06ad80 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.622+0000 7f0fb098c700 1 -- 172.21.15.38:0/3398747192 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa8011bf0 con 0x7f0fac06ad80 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/3398747192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 msgr2=0x7f0fac101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:19.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 --2- 172.21.15.38:0/3398747192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac101040 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0fa800b5f0 tx=0x7f0fa8006ee0).stop 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/3398747192 shutdown_connections 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 --2- 172.21.15.38:0/3398747192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac101040 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/3398747192 >> 172.21.15.38:0/3398747192 conn(0x7f0fac0f6680 msgr2=0x7f0fac0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/3398747192 shutdown_connections 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.623+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/3398747192 wait complete. 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb3bf2700 1 Processor -- start 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb3bf2700 1 -- start start 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb3bf2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac10cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb3bf2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac10d400 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb198e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac10cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb198e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac10cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45766/0 (socket says 172.21.15.38:45766) 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb198e700 1 -- 172.21.15.38:0/2106551112 learned_addr learned my addr 172.21.15.38:0/2106551112 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb198e700 1 -- 172.21.15.38:0/2106551112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa800b200 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.624+0000 7f0fb198e700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac10cf10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0fa8006800 tx=0x7f0fa8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.625+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa800dec0 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.625+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac10d600 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.625+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0fa801b8a0 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.625+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa801a7b0 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.625+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fac10da90 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.626+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fac045bb0 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.626+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f0fa802b070 con 0x7f0fac06ad80 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.627+0000 7f0f9a7fc700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f0f94030fa0 0x7f0f94033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.627+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0fa8043060 con 0x7f0fac06ad80 2024-02-20T13:17:19.630 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.627+0000 7f0fb118d700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f0f94030fa0 0x7f0f94033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:19.630 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.627+0000 7f0fb118d700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f0f94030fa0 0x7f0f94033460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f0f9c0096a0 tx=0x7f0f9c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:19.632 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.630+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0fa801ba10 con 0x7f0fac06ad80 2024-02-20T13:17:19.793 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.791+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0fac1020e0 con 0x7f0fac06ad80 2024-02-20T13:17:19.798 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.796+0000 7f0f9a7fc700 1 -- 172.21.15.38:0/2106551112 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0fa80330c0 con 0x7f0fac06ad80 2024-02-20T13:17:19.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f0f94030fa0 msgr2=0x7f0f94033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:19.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f0f94030fa0 0x7f0f94033460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f0f9c0096a0 tx=0x7f0f9c006b40).stop 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 msgr2=0x7f0fac10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac10cf10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0fa8006800 tx=0x7f0fa8006ee0).stop 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 shutdown_connections 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f0f94030fa0 0x7f0f94033460 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 --2- 172.21.15.38:0/2106551112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fac06ad80 0x7f0fac10cf10 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0x7f0fa8006800 tx=0x7f0fa8006ee0).stop 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 >> 172.21.15.38:0/2106551112 conn(0x7f0fac0f6680 msgr2=0x7f0fac0f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.799+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 shutdown_connections 2024-02-20T13:17:19.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:19.800+0000 7f0fb3bf2700 1 -- 172.21.15.38:0/2106551112 wait complete. 2024-02-20T13:17:20.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:19.058055+0000 mgr.smithi038.powepi (mgr.14116) 25 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:20.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:19.798047+0000 mon.smithi038 (mon.0) 68 : audit [INF] from='client.? 172.21.15.38:0/2106551112' entity='client.admin' 2024-02-20T13:17:20.506 INFO:teuthology.orchestra.run.smithi038.stderr:Enabling the dashboard module... 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.106+0000 7f4ab9a49700 1 Processor -- start 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab9a49700 1 -- start start 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab9a49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab40fda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab9a49700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40fdf50 con 0x7f4ab40fd640 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab40fda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab40fda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45780/0 (socket says 172.21.15.38:45780) 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab2ffd700 1 -- 172.21.15.38:0/2237742451 learned_addr learned my addr 172.21.15.38:0/2237742451 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab2ffd700 1 -- 172.21.15.38:0/2237742451 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab40fe090 con 0x7f4ab40fd640 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.107+0000 7f4ab2ffd700 1 --2- 172.21.15.38:0/2237742451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab40fda60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4aa400b5f0 tx=0x7f4aa4006ee0).ready entity=mon.0 client_cookie=ba54e390ccffc73c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab1ffb700 1 -- 172.21.15.38:0/2237742451 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa400d600 con 0x7f4ab40fd640 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab1ffb700 1 -- 172.21.15.38:0/2237742451 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4aa400dbf0 con 0x7f4ab40fd640 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab1ffb700 1 -- 172.21.15.38:0/2237742451 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa4011bf0 con 0x7f4ab40fd640 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 -- 172.21.15.38:0/2237742451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 msgr2=0x7f4ab40fda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 --2- 172.21.15.38:0/2237742451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab40fda60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4aa400b5f0 tx=0x7f4aa4006ee0).stop 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 -- 172.21.15.38:0/2237742451 shutdown_connections 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 --2- 172.21.15.38:0/2237742451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab40fda60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 -- 172.21.15.38:0/2237742451 >> 172.21.15.38:0/2237742451 conn(0x7f4ab40f5e20 msgr2=0x7f4ab40f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 -- 172.21.15.38:0/2237742451 shutdown_connections 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.108+0000 7f4ab9a49700 1 -- 172.21.15.38:0/2237742451 wait complete. 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.109+0000 7f4ab9a49700 1 Processor -- start 2024-02-20T13:17:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.109+0000 7f4ab9a49700 1 -- start start 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.109+0000 7f4ab9a49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab410ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.109+0000 7f4ab9a49700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab410b280 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab410ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab410ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45792/0 (socket says 172.21.15.38:45792) 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab2ffd700 1 -- 172.21.15.38:0/881633069 learned_addr learned my addr 172.21.15.38:0/881633069 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab2ffd700 1 -- 172.21.15.38:0/881633069 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa400b200 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab2ffd700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab410ad90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4aa4006800 tx=0x7f4aa4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa400dec0 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4aa401b8a0 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab410b480 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa401a7b0 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.110+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab410c0c0 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.111+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f4aa402b070 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.111+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab410b910 con 0x7f4ab40fd640 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.111+0000 7f4aab7fe700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4a94030ff0 0x7f4a940334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.111+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f4aa4042b30 con 0x7f4ab40fd640 2024-02-20T13:17:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.112+0000 7f4ab27fc700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4a94030ff0 0x7f4a940334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.112+0000 7f4ab27fc700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4a94030ff0 0x7f4a940334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f4a9c0096a0 tx=0x7f4a9c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.116+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4aa4044e60 con 0x7f4ab40fd640 2024-02-20T13:17:21.321 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:21.319+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4ab410cef0 con 0x7f4ab40fd640 2024-02-20T13:17:22.309 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.307+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f4aa400f070 con 0x7f4ab40fd640 2024-02-20T13:17:22.310 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.309+0000 7f4aab7fe700 1 -- 172.21.15.38:0/881633069 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f4aa4019d40 con 0x7f4ab40fd640 2024-02-20T13:17:22.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4a94030ff0 msgr2=0x7f4a940334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:22.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4a94030ff0 0x7f4a940334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f4a9c0096a0 tx=0x7f4a9c006b40).stop 2024-02-20T13:17:22.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 msgr2=0x7f4ab410ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:22.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab410ad90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4aa4006800 tx=0x7f4aa4006ee0).stop 2024-02-20T13:17:22.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 shutdown_connections 2024-02-20T13:17:22.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4a94030ff0 0x7f4a940334b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:22.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 --2- 172.21.15.38:0/881633069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab40fd640 0x7f4ab410ad90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:22.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 >> 172.21.15.38:0/881633069 conn(0x7f4ab40f5e20 msgr2=0x7f4ab40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:22.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 shutdown_connections 2024-02-20T13:17:22.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:22.312+0000 7f4ab9a49700 1 -- 172.21.15.38:0/881633069 wait complete. 2024-02-20T13:17:22.687 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:21.058310+0000 mgr.smithi038.powepi (mgr.14116) 26 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:22.687 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:21.321106+0000 mon.smithi038 (mon.0) 69 : audit [INF] from='client.? 172.21.15.38:0/881633069' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-20T13:17:23.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.554+0000 7f44a1eb3700 1 Processor -- start 2024-02-20T13:17:23.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.554+0000 7f44a1eb3700 1 -- start start 2024-02-20T13:17:23.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.554+0000 7f44a1eb3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:23.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.554+0000 7f44a1eb3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0ff0d0 con 0x7f449c027670 2024-02-20T13:17:23.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:23.559 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45816/0 (socket says 172.21.15.38:45816) 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449b7fe700 1 -- 172.21.15.38:0/2539033104 learned_addr learned my addr 172.21.15.38:0/2539033104 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449b7fe700 1 -- 172.21.15.38:0/2539033104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0ff210 con 0x7f449c027670 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449b7fe700 1 --2- 172.21.15.38:0/2539033104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c0febe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f448c00b5f0 tx=0x7f448c006ee0).ready entity=mon.0 client_cookie=5d6d2ca35f67f831 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449a7fc700 1 -- 172.21.15.38:0/2539033104 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f448c00d600 con 0x7f449c027670 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449a7fc700 1 -- 172.21.15.38:0/2539033104 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f448c00dbf0 con 0x7f449c027670 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.555+0000 7f449a7fc700 1 -- 172.21.15.38:0/2539033104 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f448c011bf0 con 0x7f449c027670 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 -- 172.21.15.38:0/2539033104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 msgr2=0x7f449c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 --2- 172.21.15.38:0/2539033104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c0febe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f448c00b5f0 tx=0x7f448c006ee0).stop 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 -- 172.21.15.38:0/2539033104 shutdown_connections 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 --2- 172.21.15.38:0/2539033104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c0febe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 -- 172.21.15.38:0/2539033104 >> 172.21.15.38:0/2539033104 conn(0x7f449c0f6680 msgr2=0x7f449c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 -- 172.21.15.38:0/2539033104 shutdown_connections 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.556+0000 7f44a1eb3700 1 -- 172.21.15.38:0/2539033104 wait complete. 2024-02-20T13:17:23.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f44a1eb3700 1 Processor -- start 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f44a1eb3700 1 -- start start 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f44a1eb3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c10ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f44a1eb3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c10b250 con 0x7f449c027670 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f449b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c10ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f449b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c10ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45828/0 (socket says 172.21.15.38:45828) 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.557+0000 7f449b7fe700 1 -- 172.21.15.38:0/3950223604 learned_addr learned my addr 172.21.15.38:0/3950223604 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f449b7fe700 1 -- 172.21.15.38:0/3950223604 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c00b200 con 0x7f449c027670 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f449b7fe700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c10ad60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f448c011de0 tx=0x7f448c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:23.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f448c01b410 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f448c01ba00 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f448c01a770 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c10b450 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.558+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c10c060 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.559+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f448c02b070 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.559+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c0591e0 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.559+0000 7f4483fff700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 0x7f4488033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.559+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f448c042c80 con 0x7f449c027670 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.559+0000 7f449affd700 1 -- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 msgr2=0x7f4488033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4148388115 2024-02-20T13:17:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.559+0000 7f449affd700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 0x7f4488033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:17:23.565 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.564+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f448c02b8f0 con 0x7f449c027670 2024-02-20T13:17:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:22.310154+0000 mon.smithi038 (mon.0) 70 : audit [INF] from='client.? 172.21.15.38:0/881633069' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-20T13:17:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:22.310211+0000 mon.smithi038 (mon.0) 71 : cluster [DBG] mgrmap e9: smithi038.powepi(active, since 25s) 2024-02-20T13:17:23.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.760+0000 7f449affd700 1 -- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 msgr2=0x7f4488033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4148388115 2024-02-20T13:17:23.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.760+0000 7f449affd700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 0x7f4488033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.813+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f449c100050 con 0x7f449c027670 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.814+0000 7f4483fff700 1 -- 172.21.15.38:0/3950223604 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f448c01f020 con 0x7f449c027670 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": true, 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "active_name": "smithi038.powepi", 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T13:17:23.816 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 msgr2=0x7f4488033500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 0x7f4488033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 msgr2=0x7f449c10ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c10ad60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f448c011de0 tx=0x7f448c006ee0).stop 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 shutdown_connections 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7f4488031040 0x7f4488033500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 --2- 172.21.15.38:0/3950223604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f449c027670 0x7f449c10ad60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:23.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 >> 172.21.15.38:0/3950223604 conn(0x7f449c0f6680 msgr2=0x7f449c0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:23.819 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 shutdown_connections 2024-02-20T13:17:23.819 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:23.816+0000 7f44a1eb3700 1 -- 172.21.15.38:0/3950223604 wait complete. 2024-02-20T13:17:24.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:23.815504+0000 mon.smithi038 (mon.0) 72 : audit [DBG] from='client.? 172.21.15.38:0/3950223604' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T13:17:24.523 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for the mgr to restart... 2024-02-20T13:17:24.524 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr epoch 9... 2024-02-20T13:17:25.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.120+0000 7ff0468fd700 1 Processor -- start 2024-02-20T13:17:25.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.120+0000 7ff0468fd700 1 -- start start 2024-02-20T13:17:25.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff0468fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:25.126 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff0468fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400ff130 con 0x7ff040027610 2024-02-20T13:17:25.127 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45830/0 (socket says 172.21.15.38:45830) 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03ffff700 1 -- 172.21.15.38:0/1799958282 learned_addr learned my addr 172.21.15.38:0/1799958282 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03ffff700 1 -- 172.21.15.38:0/1799958282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400ff270 con 0x7ff040027610 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03ffff700 1 --2- 172.21.15.38:0/1799958282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fec40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff03000b5f0 tx=0x7ff030006ee0).ready entity=mon.0 client_cookie=20dd03b718ca5ce8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03effd700 1 -- 172.21.15.38:0/1799958282 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03000d600 con 0x7ff040027610 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03effd700 1 -- 172.21.15.38:0/1799958282 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff03000dbf0 con 0x7ff040027610 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.121+0000 7ff03effd700 1 -- 172.21.15.38:0/1799958282 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff030011bf0 con 0x7ff040027610 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 -- 172.21.15.38:0/1799958282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 msgr2=0x7ff0400fec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 --2- 172.21.15.38:0/1799958282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fec40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff03000b5f0 tx=0x7ff030006ee0).stop 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 -- 172.21.15.38:0/1799958282 shutdown_connections 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 --2- 172.21.15.38:0/1799958282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fec40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 -- 172.21.15.38:0/1799958282 >> 172.21.15.38:0/1799958282 conn(0x7ff0400f6680 msgr2=0x7ff0400f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 -- 172.21.15.38:0/1799958282 shutdown_connections 2024-02-20T13:17:25.128 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.122+0000 7ff0468fd700 1 -- 172.21.15.38:0/1799958282 wait complete. 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff0468fd700 1 Processor -- start 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff0468fd700 1 -- start start 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff0468fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff0468fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400fab40 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff03ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff03ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45838/0 (socket says 172.21.15.38:45838) 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff03ffff700 1 -- 172.21.15.38:0/685028438 learned_addr learned my addr 172.21.15.38:0/685028438 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.123+0000 7ff03ffff700 1 -- 172.21.15.38:0/685028438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03000b200 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.124+0000 7ff03ffff700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fc6c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff030011de0 tx=0x7ff030006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.124+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03001b410 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.124+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff03001ba00 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.124+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0400face0 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.124+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03001a720 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.124+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0400fb170 con 0x7ff040027610 2024-02-20T13:17:25.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.125+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7ff03002b070 con 0x7ff040027610 2024-02-20T13:17:25.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.125+0000 7ff03cff9700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:25.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.125+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7ff030042cb0 con 0x7ff040027610 2024-02-20T13:17:25.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.125+0000 7ff03f7fe700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 msgr2=0x7ff02c033550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4148388115 2024-02-20T13:17:25.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.125+0000 7ff03f7fe700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T13:17:25.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.127+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 --> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0400591e0 con 0x7ff02c031090 2024-02-20T13:17:25.327 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.326+0000 7ff03f7fe700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 msgr2=0x7ff02c033550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4148388115 2024-02-20T13:17:25.328 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.326+0000 7ff03f7fe700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T13:17:25.728 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.727+0000 7ff03f7fe700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 msgr2=0x7ff02c033550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4148388115 2024-02-20T13:17:25.728 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:25.727+0000 7ff03f7fe700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T13:17:26.529 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:26.528+0000 7ff03f7fe700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 msgr2=0x7ff02c033550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4148388115 2024-02-20T13:17:26.530 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:26.528+0000 7ff03f7fe700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T13:17:28.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:28.129+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 10) v1 ==== 36958+0+0 (secure 0 0 0) 0x7ff03000f070 con 0x7ff040027610 2024-02-20T13:17:28.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:28.129+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 msgr2=0x7ff02c033550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T13:17:28.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:28.129+0000 7ff03cff9700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:17:28.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:27.650536+0000 mon.smithi038 (mon.0) 73 : cluster [INF] Active manager daemon smithi038.powepi restarted 2024-02-20T13:17:28.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:27.651169+0000 mon.smithi038 (mon.0) 74 : cluster [INF] Activating manager daemon smithi038.powepi 2024-02-20T13:17:28.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:27.653015+0000 mon.smithi038 (mon.0) 75 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-20T13:17:29.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.130+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7ff03001bb70 con 0x7ff040027610 2024-02-20T13:17:29.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.130+0000 7ff03cff9700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff02c034a30 0x7ff02c036e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:29.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.131+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0400591e0 con 0x7ff02c034a30 2024-02-20T13:17:29.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.131+0000 7ff03f7fe700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff02c034a30 0x7ff02c036e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:29.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.131+0000 7ff03f7fe700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff02c034a30 0x7ff02c036e20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff0340036d0 tx=0x7ff034006d60).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:29.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.132+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ff0400591e0 con 0x7ff02c034a30 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.138+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff040045bb0 con 0x7ff02c034a30 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.138+0000 7ff03cff9700 1 -- 172.21.15.38:0/685028438 <== mgr.14156 v2:172.21.15.38:6800/175894382 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff040045bb0 con 0x7ff02c034a30 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff02c034a30 msgr2=0x7ff02c036e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff02c034a30 0x7ff02c036e20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff0340036d0 tx=0x7ff034006d60).stop 2024-02-20T13:17:29.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 msgr2=0x7ff0400fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fc6c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff030011de0 tx=0x7ff030006ee0).stop 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 shutdown_connections 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff02c034a30 0x7ff02c036e20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff040027610 0x7ff0400fc6c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 --2- 172.21.15.38:0/685028438 >> [v2:172.21.15.38:6800/4148388115,v1:172.21.15.38:6801/4148388115] conn(0x7ff02c031090 0x7ff02c033550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 >> 172.21.15.38:0/685028438 conn(0x7ff0400f6680 msgr2=0x7ff0400f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 shutdown_connections 2024-02-20T13:17:29.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:29.139+0000 7ff0468fd700 1 -- 172.21.15.38:0/685028438 wait complete. 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:28.131845+0000 mon.smithi038 (mon.0) 76 : cluster [DBG] mgrmap e10: smithi038.powepi(active, starting, since 0.480763s) 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.133157+0000 mon.smithi038 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.133990+0000 mon.smithi038 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mgr metadata", "who": "smithi038.powepi", "id": "smithi038.powepi"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.135117+0000 mon.smithi038 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.135372+0000 mon.smithi038 (mon.0) 80 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.135569+0000 mon.smithi038 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:28.137794+0000 mon.smithi038 (mon.0) 82 : cluster [INF] Manager daemon smithi038.powepi is now available 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.250062+0000 mon.smithi038 (mon.0) 83 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.252521+0000 mon.smithi038 (mon.0) 84 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.272807+0000 mon.smithi038 (mon.0) 85 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.274388+0000 mon.smithi038 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:17:29.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:28.879752+0000 mon.smithi038 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:30.445 INFO:teuthology.orchestra.run.smithi038.stderr:mgr epoch 9 is available 2024-02-20T13:17:30.445 INFO:teuthology.orchestra.run.smithi038.stderr:Generating a dashboard self-signed certificate... 2024-02-20T13:17:30.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:30 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:29.133233+0000 mgr.smithi038.powepi (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T13:17:30.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:30 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:29.133757+0000 mon.smithi038 (mon.0) 88 : cluster [DBG] mgrmap e11: smithi038.powepi(active, since 1.48268s) 2024-02-20T13:17:30.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:30 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:29.139901+0000 mgr.smithi038.powepi (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T13:17:31.209 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:30.143404+0000 mon.smithi038 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi038.powepi(active, since 2s) 2024-02-20T13:17:31.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.588+0000 7f3b5c5f5700 1 Processor -- start 2024-02-20T13:17:31.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.588+0000 7f3b5c5f5700 1 -- start start 2024-02-20T13:17:31.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5c5f5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b540fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:31.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5c5f5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b540fcec0 con 0x7f3b54027670 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5a391700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b540fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5a391700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b540fc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38086/0 (socket says 172.21.15.38:38086) 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5a391700 1 -- 172.21.15.38:0/1269763281 learned_addr learned my addr 172.21.15.38:0/1269763281 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5a391700 1 -- 172.21.15.38:0/1269763281 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b540fd000 con 0x7f3b54027670 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5a391700 1 --2- 172.21.15.38:0/1269763281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b540fc9d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3b4400b5f0 tx=0x7f3b44006ee0).ready entity=mon.0 client_cookie=cddf24be32513c95 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5938f700 1 -- 172.21.15.38:0/1269763281 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4400d600 con 0x7f3b54027670 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5938f700 1 -- 172.21.15.38:0/1269763281 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3b4400dbf0 con 0x7f3b54027670 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.589+0000 7f3b5938f700 1 -- 172.21.15.38:0/1269763281 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b44011bf0 con 0x7f3b54027670 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/1269763281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 msgr2=0x7f3b540fc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 --2- 172.21.15.38:0/1269763281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b540fc9d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3b4400b5f0 tx=0x7f3b44006ee0).stop 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/1269763281 shutdown_connections 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 --2- 172.21.15.38:0/1269763281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b540fc9d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/1269763281 >> 172.21.15.38:0/1269763281 conn(0x7f3b540f6680 msgr2=0x7f3b540f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/1269763281 shutdown_connections 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.590+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/1269763281 wait complete. 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5c5f5700 1 Processor -- start 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5c5f5700 1 -- start start 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5c5f5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b54111330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5c5f5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54111820 con 0x7f3b54027670 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5a391700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b54111330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5a391700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b54111330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38094/0 (socket says 172.21.15.38:38094) 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5a391700 1 -- 172.21.15.38:0/2553741429 learned_addr learned my addr 172.21.15.38:0/2553741429 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5a391700 1 -- 172.21.15.38:0/2553741429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4400b200 con 0x7f3b54027670 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5a391700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b54111330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3b44011de0 tx=0x7f3b44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4401b410 con 0x7f3b54027670 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3b4401ba00 con 0x7f3b54027670 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b54111a20 con 0x7f3b54027670 2024-02-20T13:17:31.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4401a770 con 0x7f3b54027670 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.591+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b54111eb0 con 0x7f3b54027670 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.592+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f3b4402b070 con 0x7f3b54027670 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.592+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b54045bb0 con 0x7f3b54027670 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.593+0000 7f3b42ffd700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3b3c031040 0x7f3b3c033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.593+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3b44042cd0 con 0x7f3b54027670 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.593+0000 7f3b59b90700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3b3c031040 0x7f3b3c033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.593+0000 7f3b59b90700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3b3c031040 0x7f3b3c033500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3b500096a0 tx=0x7f3b50006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.598+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3b4401b570 con 0x7f3b54027670 2024-02-20T13:17:31.764 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.763+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b54112fb0 con 0x7f3b3c031040 2024-02-20T13:17:31.846 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b42ffd700 1 -- 172.21.15.38:0/2553741429 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3b54112fb0 con 0x7f3b3c031040 2024-02-20T13:17:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3b3c031040 msgr2=0x7f3b3c033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3b3c031040 0x7f3b3c033500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3b500096a0 tx=0x7f3b50006b40).stop 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 msgr2=0x7f3b54111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b54111330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3b44011de0 tx=0x7f3b44006ee0).stop 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 shutdown_connections 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3b3c031040 0x7f3b3c033500 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.846+0000 7f3b5c5f5700 1 --2- 172.21.15.38:0/2553741429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b54027670 0x7f3b54111330 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.847+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 >> 172.21.15.38:0/2553741429 conn(0x7f3b540f6680 msgr2=0x7f3b540f8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.847+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 shutdown_connections 2024-02-20T13:17:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:31.847+0000 7f3b5c5f5700 1 -- 172.21.15.38:0/2553741429 wait complete. 2024-02-20T13:17:32.554 INFO:teuthology.orchestra.run.smithi038.stderr:Creating initial admin user... 2024-02-20T13:17:33.119 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.116+0000 7fe3005f2700 1 Processor -- start 2024-02-20T13:17:33.119 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.116+0000 7fe3005f2700 1 -- start start 2024-02-20T13:17:33.119 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.117+0000 7fe3005f2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f8070100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.117+0000 7fe3005f2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f8072020 con 0x7fe2f8071c00 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.117+0000 7fe2fe38e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f8070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.117+0000 7fe2fe38e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f8070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38096/0 (socket says 172.21.15.38:38096) 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.117+0000 7fe2fe38e700 1 -- 172.21.15.38:0/2266075197 learned_addr learned my addr 172.21.15.38:0/2266075197 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.117+0000 7fe2fe38e700 1 -- 172.21.15.38:0/2266075197 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f80705f0 con 0x7fe2f8071c00 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe2fe38e700 1 --2- 172.21.15.38:0/2266075197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f8070100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe2f400b5f0 tx=0x7fe2f4006ee0).ready entity=mon.0 client_cookie=e1c1d7d7737f22da server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe2fd38c700 1 -- 172.21.15.38:0/2266075197 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f400d600 con 0x7fe2f8071c00 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe2fd38c700 1 -- 172.21.15.38:0/2266075197 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2f400dbf0 con 0x7fe2f8071c00 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe2fd38c700 1 -- 172.21.15.38:0/2266075197 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f4011bf0 con 0x7fe2f8071c00 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe3005f2700 1 -- 172.21.15.38:0/2266075197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 msgr2=0x7fe2f8070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe3005f2700 1 --2- 172.21.15.38:0/2266075197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f8070100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe2f400b5f0 tx=0x7fe2f4006ee0).stop 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe3005f2700 1 -- 172.21.15.38:0/2266075197 shutdown_connections 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe3005f2700 1 --2- 172.21.15.38:0/2266075197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f8070100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.118+0000 7fe3005f2700 1 -- 172.21.15.38:0/2266075197 >> 172.21.15.38:0/2266075197 conn(0x7fe2f80f7770 msgr2=0x7fe2f80f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.119+0000 7fe3005f2700 1 -- 172.21.15.38:0/2266075197 shutdown_connections 2024-02-20T13:17:33.121 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.119+0000 7fe3005f2700 1 -- 172.21.15.38:0/2266075197 wait complete. 2024-02-20T13:17:33.122 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.119+0000 7fe3005f2700 1 Processor -- start 2024-02-20T13:17:33.122 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.119+0000 7fe3005f2700 1 -- start start 2024-02-20T13:17:33.122 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.119+0000 7fe3005f2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:33.122 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe3005f2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f810d4d0 con 0x7fe2f8071c00 2024-02-20T13:17:33.122 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2fe38e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2fe38e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f810cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38104/0 (socket says 172.21.15.38:38104) 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2fe38e700 1 -- 172.21.15.38:0/2224148491 learned_addr learned my addr 172.21.15.38:0/2224148491 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2fe38e700 1 -- 172.21.15.38:0/2224148491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f400b200 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2fe38e700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f810cfe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe2f4006800 tx=0x7fe2f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f400dec0 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2f401b8a0 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f401a700 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f810d6d0 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.120+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f810db60 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.121+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fe2f402b070 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.121+0000 7fe2e6ffd700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe2e0031040 0x7fe2e0033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.121+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2f4042cb0 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.121+0000 7fe2fdb8d700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe2e0031040 0x7fe2e0033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.122+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f8045bb0 con 0x7fe2f8071c00 2024-02-20T13:17:33.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.122+0000 7fe2fdb8d700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe2e0031040 0x7fe2e0033500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe2e80096a0 tx=0x7fe2e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:33.126 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.126+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe2f401f080 con 0x7fe2f8071c00 2024-02-20T13:17:33.195 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:31.764846+0000 mgr.smithi038.powepi (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:33.195 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:31.841274+0000 mon.smithi038 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:33.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:31.846254+0000 mon.smithi038 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:33.304 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.303+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe2f81021e0 con 0x7fe2e0031040 2024-02-20T13:17:33.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.559+0000 7fe2e6ffd700 1 -- 172.21.15.38:0/2224148491 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fe2e80092a0 con 0x7fe2e0031040 2024-02-20T13:17:33.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$jOb0okAyQo.BXJE/Sw99xOfLhZ/JgCmIDuupx04QpgeCQyqDufglu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708435053, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-20T13:17:33.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe2e0031040 msgr2=0x7fe2e0033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:33.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe2e0031040 0x7fe2e0033500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe2e80096a0 tx=0x7fe2e8006b40).stop 2024-02-20T13:17:33.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 msgr2=0x7fe2f810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:33.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f810cfe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe2f4006800 tx=0x7fe2f4006ee0).stop 2024-02-20T13:17:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 shutdown_connections 2024-02-20T13:17:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe2e0031040 0x7fe2e0033500 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 --2- 172.21.15.38:0/2224148491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2f8071c00 0x7fe2f810cfe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 >> 172.21.15.38:0/2224148491 conn(0x7fe2f80f7770 msgr2=0x7fe2f80f9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 shutdown_connections 2024-02-20T13:17:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:33.561+0000 7fe3005f2700 1 -- 172.21.15.38:0/2224148491 wait complete. 2024-02-20T13:17:33.841 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:33.305103+0000 mgr.smithi038.powepi (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:34.195 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:33.559590+0000 mon.smithi038 (mon.0) 92 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:34.196 INFO:teuthology.orchestra.run.smithi038.stderr:Fetching dashboard port number... 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23ddcc9700 1 Processor -- start 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23ddcc9700 1 -- start start 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23ddcc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d80fc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23ddcc9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80fc770 con 0x7f23d80fbe60 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23dccc7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d80fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23dccc7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d80fc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38110/0 (socket says 172.21.15.38:38110) 2024-02-20T13:17:34.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.720+0000 7f23dccc7700 1 -- 172.21.15.38:0/3282255462 learned_addr learned my addr 172.21.15.38:0/3282255462 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23dccc7700 1 -- 172.21.15.38:0/3282255462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80fc8b0 con 0x7f23d80fbe60 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23dccc7700 1 --2- 172.21.15.38:0/3282255462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d80fc280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f23cc00b5f0 tx=0x7f23cc006ee0).ready entity=mon.0 client_cookie=e04139f2b3858fc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23d77fe700 1 -- 172.21.15.38:0/3282255462 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23cc00d600 con 0x7f23d80fbe60 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23d77fe700 1 -- 172.21.15.38:0/3282255462 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f23cc00dbf0 con 0x7f23d80fbe60 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23d77fe700 1 -- 172.21.15.38:0/3282255462 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23cc011bf0 con 0x7f23d80fbe60 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 -- 172.21.15.38:0/3282255462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 msgr2=0x7f23d80fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 --2- 172.21.15.38:0/3282255462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d80fc280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f23cc00b5f0 tx=0x7f23cc006ee0).stop 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 -- 172.21.15.38:0/3282255462 shutdown_connections 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 --2- 172.21.15.38:0/3282255462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d80fc280 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 -- 172.21.15.38:0/3282255462 >> 172.21.15.38:0/3282255462 conn(0x7f23d80f7600 msgr2=0x7f23d80f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 -- 172.21.15.38:0/3282255462 shutdown_connections 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.721+0000 7f23ddcc9700 1 -- 172.21.15.38:0/3282255462 wait complete. 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.722+0000 7f23ddcc9700 1 Processor -- start 2024-02-20T13:17:34.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.722+0000 7f23ddcc9700 1 -- start start 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23ddcc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d810aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23ddcc9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d810afc0 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23dccc7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d810aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23dccc7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d810aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38118/0 (socket says 172.21.15.38:38118) 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23dccc7700 1 -- 172.21.15.38:0/1865081720 learned_addr learned my addr 172.21.15.38:0/1865081720 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23dccc7700 1 -- 172.21.15.38:0/1865081720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc00b200 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23dccc7700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d810aad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f23cc011e30 tx=0x7f23cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23cc01b410 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d810b1c0 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f23cc01ba00 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23cc01a7b0 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.723+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d810b650 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.724+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f23cc02b070 con 0x7f23d80fbe60 2024-02-20T13:17:34.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.724+0000 7f23d57fa700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f23bc031090 0x7f23bc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:34.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.724+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f23cc042dd0 con 0x7f23d80fbe60 2024-02-20T13:17:34.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.724+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d8045bb0 con 0x7f23d80fbe60 2024-02-20T13:17:34.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.724+0000 7f23d7fff700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f23bc031090 0x7f23bc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:34.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.725+0000 7f23d7fff700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f23bc031090 0x7f23bc033550 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f23c40096a0 tx=0x7f23c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:34.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.729+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f23cc016070 con 0x7f23d80fbe60 2024-02-20T13:17:34.889 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.888+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f23d80fdad0 con 0x7f23d80fbe60 2024-02-20T13:17:34.889 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.888+0000 7f23d57fa700 1 -- 172.21.15.38:0/1865081720 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f23cc01f030 con 0x7f23d80fbe60 2024-02-20T13:17:34.889 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 8443 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f23bc031090 msgr2=0x7f23bc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f23bc031090 0x7f23bc033550 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f23c40096a0 tx=0x7f23c4006b40).stop 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 msgr2=0x7f23d810aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d810aad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f23cc011e30 tx=0x7f23cc006ee0).stop 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 shutdown_connections 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f23bc031090 0x7f23bc033550 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 --2- 172.21.15.38:0/1865081720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23d80fbe60 0x7f23d810aad0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:34.891 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 >> 172.21.15.38:0/1865081720 conn(0x7f23d80f7600 msgr2=0x7f23d80f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:34.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 shutdown_connections 2024-02-20T13:17:34.892 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:34.890+0000 7f23ddcc9700 1 -- 172.21.15.38:0/1865081720 wait complete. 2024-02-20T13:17:35.556 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2024-02-20T13:17:35.565 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2024-02-20T13:17:35.565 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2024-02-20T13:17:35.566 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr:Ceph Dashboard is now available at: 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr: URL: https://smithi038.front.sepia.ceph.com:8443/ 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr: User: admin 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr: Password: 2fayveo3hd 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:35.569 INFO:teuthology.orchestra.run.smithi038.stderr:Enabling autotune for osd_memory_target 2024-02-20T13:17:35.804 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:34.566391+0000 mon.smithi038 (mon.0) 93 : cluster [DBG] mgrmap e13: smithi038.powepi(active, since 6s) 2024-02-20T13:17:35.804 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:34.888679+0000 mon.smithi038 (mon.0) 94 : audit [DBG] from='client.? 172.21.15.38:0/1865081720' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-20T13:17:36.186 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.184+0000 7f919175c700 1 Processor -- start 2024-02-20T13:17:36.186 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.184+0000 7f919175c700 1 -- start start 2024-02-20T13:17:36.186 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.185+0000 7f919175c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c0fc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:36.186 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.185+0000 7f919175c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0fc8f0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.186 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.185+0000 7f918affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c0fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.185+0000 7f918affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c0fc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38124/0 (socket says 172.21.15.38:38124) 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.185+0000 7f918affd700 1 -- 172.21.15.38:0/30847898 learned_addr learned my addr 172.21.15.38:0/30847898 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.185+0000 7f918affd700 1 -- 172.21.15.38:0/30847898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0fca30 con 0x7f918c0fbfe0 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f918affd700 1 --2- 172.21.15.38:0/30847898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c0fc400 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f918000b5f0 tx=0x7f9180006ee0).ready entity=mon.0 client_cookie=72b14ad2608b163d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f9189ffb700 1 -- 172.21.15.38:0/30847898 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f918000d600 con 0x7f918c0fbfe0 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f9189ffb700 1 -- 172.21.15.38:0/30847898 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f918000dbf0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.187 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f9189ffb700 1 -- 172.21.15.38:0/30847898 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9180011bf0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.189 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 -- 172.21.15.38:0/30847898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 msgr2=0x7f918c0fc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:36.189 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 --2- 172.21.15.38:0/30847898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c0fc400 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f918000b5f0 tx=0x7f9180006ee0).stop 2024-02-20T13:17:36.189 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 -- 172.21.15.38:0/30847898 shutdown_connections 2024-02-20T13:17:36.189 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 --2- 172.21.15.38:0/30847898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c0fc400 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 -- 172.21.15.38:0/30847898 >> 172.21.15.38:0/30847898 conn(0x7f918c0f7790 msgr2=0x7f918c0f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 -- 172.21.15.38:0/30847898 shutdown_connections 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.186+0000 7f919175c700 1 -- 172.21.15.38:0/30847898 wait complete. 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.187+0000 7f919175c700 1 Processor -- start 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.187+0000 7f919175c700 1 -- start start 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.187+0000 7f919175c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.187+0000 7f919175c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0700e0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f918affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f918affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38128/0 (socket says 172.21.15.38:38128) 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f918affd700 1 -- 172.21.15.38:0/277675887 learned_addr learned my addr 172.21.15.38:0/277675887 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f918affd700 1 -- 172.21.15.38:0/277675887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918000b200 con 0x7f918c0fbfe0 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f918affd700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c071c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9180011e00 tx=0x7f9180006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f918001b410 con 0x7f918c0fbfe0 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f918001ba00 con 0x7f918c0fbfe0 2024-02-20T13:17:36.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c070280 con 0x7f918c0fbfe0 2024-02-20T13:17:36.191 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f918001a780 con 0x7f918c0fbfe0 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.188+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c070710 con 0x7f918c0fbfe0 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.189+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f918002b070 con 0x7f918c0fbfe0 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.189+0000 7f91737fe700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f91780310e0 0x7f91780335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.189+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9180042c80 con 0x7f918c0fbfe0 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.189+0000 7f918a7fc700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f91780310e0 0x7f91780335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.190+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c0591e0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.190+0000 7f918a7fc700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f91780310e0 0x7f91780335a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f917c0096a0 tx=0x7f917c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:36.194 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.194+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9180016070 con 0x7f918c0fbfe0 2024-02-20T13:17:36.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.353+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f918c045bb0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.354+0000 7f91737fe700 1 -- 172.21.15.38:0/277675887 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v10)=0 v10) v1 ==== 127+0+0 (secure 0 0 0) 0x7f91800330c0 con 0x7f918c0fbfe0 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f91780310e0 msgr2=0x7f91780335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f91780310e0 0x7f91780335a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f917c0096a0 tx=0x7f917c006b40).stop 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 msgr2=0x7f918c071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c071c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9180011e00 tx=0x7f9180006ee0).stop 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 shutdown_connections 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f91780310e0 0x7f91780335a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 --2- 172.21.15.38:0/277675887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f918c0fbfe0 0x7f918c071c60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 >> 172.21.15.38:0/277675887 conn(0x7f918c0f7790 msgr2=0x7f918c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 shutdown_connections 2024-02-20T13:17:36.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:36.355+0000 7f919175c700 1 -- 172.21.15.38:0/277675887 wait complete. 2024-02-20T13:17:37.751 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.749+0000 7f87e4ef4700 1 Processor -- start 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.749+0000 7f87e4ef4700 1 -- start start 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87e4ef4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e0071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87e4ef4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0073090 con 0x7f87e0072c70 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e0071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e0071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58606/0 (socket says 172.21.15.38:58606) 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87de59c700 1 -- 172.21.15.38:0/3641103500 learned_addr learned my addr 172.21.15.38:0/3641103500 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87de59c700 1 -- 172.21.15.38:0/3641103500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0071700 con 0x7f87e0072c70 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.750+0000 7f87de59c700 1 --2- 172.21.15.38:0/3641103500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e0071210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f87c800b5f0 tx=0x7f87c8006ee0).ready entity=mon.0 client_cookie=5f66c95e2cc2f946 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87dd59a700 1 -- 172.21.15.38:0/3641103500 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87c800d600 con 0x7f87e0072c70 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87dd59a700 1 -- 172.21.15.38:0/3641103500 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f87c800dbf0 con 0x7f87e0072c70 2024-02-20T13:17:37.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87dd59a700 1 -- 172.21.15.38:0/3641103500 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87c8011bf0 con 0x7f87e0072c70 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 -- 172.21.15.38:0/3641103500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 msgr2=0x7f87e0071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 --2- 172.21.15.38:0/3641103500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e0071210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f87c800b5f0 tx=0x7f87c8006ee0).stop 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 -- 172.21.15.38:0/3641103500 shutdown_connections 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 --2- 172.21.15.38:0/3641103500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e0071210 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 -- 172.21.15.38:0/3641103500 >> 172.21.15.38:0/3641103500 conn(0x7f87e00f7780 msgr2=0x7f87e00f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 -- 172.21.15.38:0/3641103500 shutdown_connections 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.751+0000 7f87e4ef4700 1 -- 172.21.15.38:0/3641103500 wait complete. 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87e4ef4700 1 Processor -- start 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87e4ef4700 1 -- start start 2024-02-20T13:17:37.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87e4ef4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e010cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87e4ef4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e010d490 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e010cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e010cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58610/0 (socket says 172.21.15.38:58610) 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.752+0000 7f87de59c700 1 -- 172.21.15.38:0/931091879 learned_addr learned my addr 172.21.15.38:0/931091879 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87de59c700 1 -- 172.21.15.38:0/931091879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c800b200 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87de59c700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e010cfa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f87c8006800 tx=0x7f87c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87c800dec0 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e010d690 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f87c801b8a0 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87c801a6f0 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.753+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e010db20 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.754+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f87c802b070 con 0x7f87e0072c70 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.754+0000 7f87d6ffd700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f87c0031090 0x7f87c0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:37.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.754+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f87c8042b90 con 0x7f87e0072c70 2024-02-20T13:17:37.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.754+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e0045bb0 con 0x7f87e0072c70 2024-02-20T13:17:37.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.754+0000 7f87ddd9b700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f87c0031090 0x7f87c0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:37.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.755+0000 7f87ddd9b700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f87c0031090 0x7f87c0033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f87d00096a0 tx=0x7f87d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:37.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:37.759+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f87c801f070 con 0x7f87e0072c70 2024-02-20T13:17:38.005 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.004+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f87e0102140 con 0x7f87e0072c70 2024-02-20T13:17:38.007 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.007+0000 7f87d6ffd700 1 -- 172.21.15.38:0/931091879 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v31)=0 set mgr/dashboard/cluster/status v31) v1 ==== 153+0+0 (secure 0 0 0) 0x7f87c8016030 con 0x7f87e0072c70 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f87c0031090 msgr2=0x7f87c0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f87c0031090 0x7f87c0033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f87d00096a0 tx=0x7f87d0006b40).stop 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 msgr2=0x7f87e010cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e010cfa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f87c8006800 tx=0x7f87c8006ee0).stop 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 shutdown_connections 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f87c0031090 0x7f87c0033550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 --2- 172.21.15.38:0/931091879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e0072c70 0x7f87e010cfa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.008+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 >> 172.21.15.38:0/931091879 conn(0x7f87e00f7780 msgr2=0x7f87e00f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.009+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 shutdown_connections 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2024-02-20T13:17:38.009+0000 7f87e4ef4700 1 -- 172.21.15.38:0/931091879 wait complete. 2024-02-20T13:17:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-20T13:17:38.619 INFO:teuthology.orchestra.run.smithi038.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-20T13:17:38.619 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.619 INFO:teuthology.orchestra.run.smithi038.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 24090638-cff2-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-20T13:17:38.619 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr:Or, if you are only running a single cluster on this host: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr: ceph telemetry on 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.620 INFO:teuthology.orchestra.run.smithi038.stderr:For more information see: 2024-02-20T13:17:38.621 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.621 INFO:teuthology.orchestra.run.smithi038.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-20T13:17:38.621 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:17:38.621 INFO:teuthology.orchestra.run.smithi038.stderr:Bootstrap complete. 2024-02-20T13:17:38.661 INFO:tasks.cephadm:Fetching config... 2024-02-20T13:17:38.662 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:17:38.662 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-20T13:17:38.720 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-20T13:17:38.721 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:17:38.721 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-20T13:17:38.778 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-20T13:17:38.778 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:17:38.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/mon.smithi038/keyring of=/dev/stdout 2024-02-20T13:17:38.847 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-20T13:17:38.847 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-02-20T13:17:38.847 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-20T13:17:38.905 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-20T13:17:38.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3CKiYEjJ9Oe1YYj8RpxZWAbVJefJR5/FmzuhAFkhkPWUYDV1UlcRSTABPppqJkbOevKKNn27AIgSBZKp9vpyK7XBfnpvkWnBf6uvrmCNfBMASJgKCuUyEkzeMa0WWA1tv47FAQ3Kg9r1c7zPdc23pPErRlepbINL1SOWnXS/FoKe+7KeG0XpLqlQP4mZeNPFqp4LBP/xTOUYzhWClJ5AsQnzPD6ysQkbl6cDlaN/8o+ZopCTJpA5iB6rG9++RcclXYplZyTBflLN17LTebHVAV9XWyVLDhuRPH8BgKYUT0g7ZmJu+FumU67VsAHsG2bNhKP5X6f0EUN/4k5OUa/FpGTzQQZyeugpzgOye5gU9DHTOL33F0WcafCGRiUMRPawGFHiIdqm+Nvy/6v39Uo3Q1xiI33IYcXd51i1XOr+Lt+MDCDoNeENhToutWQJPPlqfdRdBTpXEfHEWb31POGJnXFTepEgG2P3CedbEb21758erC+fhiZu8BIyod7TlR/E= ceph-24090638-cff2-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T13:17:38.984 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3CKiYEjJ9Oe1YYj8RpxZWAbVJefJR5/FmzuhAFkhkPWUYDV1UlcRSTABPppqJkbOevKKNn27AIgSBZKp9vpyK7XBfnpvkWnBf6uvrmCNfBMASJgKCuUyEkzeMa0WWA1tv47FAQ3Kg9r1c7zPdc23pPErRlepbINL1SOWnXS/FoKe+7KeG0XpLqlQP4mZeNPFqp4LBP/xTOUYzhWClJ5AsQnzPD6ysQkbl6cDlaN/8o+ZopCTJpA5iB6rG9++RcclXYplZyTBflLN17LTebHVAV9XWyVLDhuRPH8BgKYUT0g7ZmJu+FumU67VsAHsG2bNhKP5X6f0EUN/4k5OUa/FpGTzQQZyeugpzgOye5gU9DHTOL33F0WcafCGRiUMRPawGFHiIdqm+Nvy/6v39Uo3Q1xiI33IYcXd51i1XOr+Lt+MDCDoNeENhToutWQJPPlqfdRdBTpXEfHEWb31POGJnXFTepEgG2P3CedbEb21758erC+fhiZu8BIyod7TlR/E= ceph-24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:17:38.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3CKiYEjJ9Oe1YYj8RpxZWAbVJefJR5/FmzuhAFkhkPWUYDV1UlcRSTABPppqJkbOevKKNn27AIgSBZKp9vpyK7XBfnpvkWnBf6uvrmCNfBMASJgKCuUyEkzeMa0WWA1tv47FAQ3Kg9r1c7zPdc23pPErRlepbINL1SOWnXS/FoKe+7KeG0XpLqlQP4mZeNPFqp4LBP/xTOUYzhWClJ5AsQnzPD6ysQkbl6cDlaN/8o+ZopCTJpA5iB6rG9++RcclXYplZyTBflLN17LTebHVAV9XWyVLDhuRPH8BgKYUT0g7ZmJu+FumU67VsAHsG2bNhKP5X6f0EUN/4k5OUa/FpGTzQQZyeugpzgOye5gU9DHTOL33F0WcafCGRiUMRPawGFHiIdqm+Nvy/6v39Uo3Q1xiI33IYcXd51i1XOr+Lt+MDCDoNeENhToutWQJPPlqfdRdBTpXEfHEWb31POGJnXFTepEgG2P3CedbEb21758erC+fhiZu8BIyod7TlR/E= ceph-24090638-cff2-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T13:17:39.041 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3CKiYEjJ9Oe1YYj8RpxZWAbVJefJR5/FmzuhAFkhkPWUYDV1UlcRSTABPppqJkbOevKKNn27AIgSBZKp9vpyK7XBfnpvkWnBf6uvrmCNfBMASJgKCuUyEkzeMa0WWA1tv47FAQ3Kg9r1c7zPdc23pPErRlepbINL1SOWnXS/FoKe+7KeG0XpLqlQP4mZeNPFqp4LBP/xTOUYzhWClJ5AsQnzPD6ysQkbl6cDlaN/8o+ZopCTJpA5iB6rG9++RcclXYplZyTBflLN17LTebHVAV9XWyVLDhuRPH8BgKYUT0g7ZmJu+FumU67VsAHsG2bNhKP5X6f0EUN/4k5OUa/FpGTzQQZyeugpzgOye5gU9DHTOL33F0WcafCGRiUMRPawGFHiIdqm+Nvy/6v39Uo3Q1xiI33IYcXd51i1XOr+Lt+MDCDoNeENhToutWQJPPlqfdRdBTpXEfHEWb31POGJnXFTepEgG2P3CedbEb21758erC+fhiZu8BIyod7TlR/E= ceph-24090638-cff2-11ee-95bb-87774f69a715 2024-02-20T13:17:39.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-20T13:17:39.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:38.007222+0000 mon.smithi038 (mon.0) 95 : audit [INF] from='client.? 172.21.15.38:0/931091879' entity='client.admin' 2024-02-20T13:17:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 -- 172.21.15.38:0/2594411055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 msgr2=0x7f5e5c0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 --2- 172.21.15.38:0/2594411055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f9160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5e4c009fb0 tx=0x7f5e4c009b30).stop 2024-02-20T13:17:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 -- 172.21.15.38:0/2594411055 shutdown_connections 2024-02-20T13:17:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 --2- 172.21.15.38:0/2594411055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f9160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 -- 172.21.15.38:0/2594411055 >> 172.21.15.38:0/2594411055 conn(0x7f5e5c0f0be0 msgr2=0x7f5e5c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 -- 172.21.15.38:0/2594411055 shutdown_connections 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 -- 172.21.15.38:0/2594411055 wait complete. 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.884+0000 7f5e614de700 1 Processor -- start 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e614de700 1 -- start start 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e614de700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e614de700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0f5060 con 0x7f5e5c060360 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e5affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e5affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58626/0 (socket says 172.21.15.38:58626) 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e5affd700 1 -- 172.21.15.38:0/1162124877 learned_addr learned my addr 172.21.15.38:0/1162124877 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e5affd700 1 -- 172.21.15.38:0/1162124877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c005040 con 0x7f5e5c060360 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e5affd700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f6be0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5e4c000c00 tx=0x7f5e4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e4c0095f0 con 0x7f5e5c060360 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5e4c007e50 con 0x7f5e5c060360 2024-02-20T13:17:41.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c0f5200 con 0x7f5e5c060360 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.885+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e4c014e70 con 0x7f5e5c060360 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.886+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c0f5690 con 0x7f5e5c060360 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.886+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5e4c025070 con 0x7f5e5c060360 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.886+0000 7f5e437fe700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e44031090 0x7f5e44033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.886+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5e4c042cb0 con 0x7f5e5c060360 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.886+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c0591e0 con 0x7f5e5c060360 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.887+0000 7f5e5a7fc700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e44031090 0x7f5e44033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.887+0000 7f5e5a7fc700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e44031090 0x7f5e44033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5e500096a0 tx=0x7f5e50006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:41.891+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5e4c026350 con 0x7f5e5c060360 2024-02-20T13:17:42.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.052+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5e5c045bb0 con 0x7f5e5c060360 2024-02-20T13:17:42.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.056+0000 7f5e437fe700 1 -- 172.21.15.38:0/1162124877 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5e4c02d0c0 con 0x7f5e5c060360 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e44031090 msgr2=0x7f5e44033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e44031090 0x7f5e44033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5e500096a0 tx=0x7f5e50006b40).stop 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 msgr2=0x7f5e5c0f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f6be0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5e4c000c00 tx=0x7f5e4c00b040).stop 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 shutdown_connections 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e44031090 0x7f5e44033550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:42.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.057+0000 7f5e614de700 1 --2- 172.21.15.38:0/1162124877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e5c060360 0x7f5e5c0f6be0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:42.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.058+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 >> 172.21.15.38:0/1162124877 conn(0x7f5e5c0f0be0 msgr2=0x7f5e5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:42.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.058+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 shutdown_connections 2024-02-20T13:17:42.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:42.058+0000 7f5e614de700 1 -- 172.21.15.38:0/1162124877 wait complete. 2024-02-20T13:17:42.857 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:41.485763+0000 mon.smithi038 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:42.857 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:42.056175+0000 mon.smithi038 (mon.0) 97 : audit [INF] from='client.? 172.21.15.38:0/1162124877' entity='client.admin' 2024-02-20T13:17:42.857 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:42.221993+0000 mon.smithi038 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-02-20T13:17:42.857 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:42.225220+0000 mon.smithi038 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:42.857 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:42.231896+0000 mgr.smithi038.powepi (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi038 on smithi038 2024-02-20T13:17:42.875 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-20T13:17:42.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-20T13:17:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 -- 172.21.15.38:0/810035127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 msgr2=0x7f024c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 --2- 172.21.15.38:0/810035127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c0f69b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f0240004760 tx=0x7f0240009ad0).stop 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 -- 172.21.15.38:0/810035127 shutdown_connections 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 --2- 172.21.15.38:0/810035127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c0f69b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 -- 172.21.15.38:0/810035127 >> 172.21.15.38:0/810035127 conn(0x7f024c0f1cd0 msgr2=0x7f024c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 -- 172.21.15.38:0/810035127 shutdown_connections 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.176+0000 7f0252b84700 1 -- 172.21.15.38:0/810035127 wait complete. 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0252b84700 1 Processor -- start 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0252b84700 1 -- start start 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0252b84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0252b84700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c106cb0 con 0x7f024c0f6590 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0250920700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0250920700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58642/0 (socket says 172.21.15.38:58642) 2024-02-20T13:17:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.177+0000 7f0250920700 1 -- 172.21.15.38:0/2980062701 learned_addr learned my addr 172.21.15.38:0/2980062701 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f0250920700 1 -- 172.21.15.38:0/2980062701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240009d70 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f0250920700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c1067c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0240004170 tx=0x7f0240009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f024000e040 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0240008ef0 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c106eb0 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0240014c90 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.178+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c107340 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.179+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f02400283f0 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.179+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c045bb0 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.179+0000 7f02497fa700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f023c031090 0x7f023c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.179+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0240042240 con 0x7f024c0f6590 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.179+0000 7f024bfff700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f023c031090 0x7f023c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.180+0000 7f024bfff700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f023c031090 0x7f023c033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f02380096a0 tx=0x7f0238006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.184+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f02400266c0 con 0x7f024c0f6590 2024-02-20T13:17:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.347+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f024c0f2ed0 con 0x7f023c031090 2024-02-20T13:17:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.350+0000 7f02497fa700 1 -- 172.21.15.38:0/2980062701 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f024c0f2ed0 con 0x7f023c031090 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f023c031090 msgr2=0x7f023c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f023c031090 0x7f023c033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f02380096a0 tx=0x7f0238006b40).stop 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 msgr2=0x7f024c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c1067c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0240004170 tx=0x7f0240009ad0).stop 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 shutdown_connections 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f023c031090 0x7f023c033550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 --2- 172.21.15.38:0/2980062701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f024c0f6590 0x7f024c1067c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 >> 172.21.15.38:0/2980062701 conn(0x7f024c0f1cd0 msgr2=0x7f024c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 shutdown_connections 2024-02-20T13:17:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:45.352+0000 7f0252b84700 1 -- 172.21.15.38:0/2980062701 wait complete. 2024-02-20T13:17:45.921 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2024-02-20T13:17:45.921 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:17:45.922 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2024-02-20T13:17:45.971 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-20T13:17:45.971 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T13:17:46.032 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2024-02-20T13:17:46.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph orch host add smithi114 2024-02-20T13:17:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:45.348314+0000 mgr.smithi038.powepi (mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:45.350914+0000 mon.smithi038 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 -- 172.21.15.38:0/2253492970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 msgr2=0x7f8f800f4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 --2- 172.21.15.38:0/2253492970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f800f4ab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8f70009fb0 tx=0x7f8f70009b30).stop 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 -- 172.21.15.38:0/2253492970 shutdown_connections 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 --2- 172.21.15.38:0/2253492970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f800f4ab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 -- 172.21.15.38:0/2253492970 >> 172.21.15.38:0/2253492970 conn(0x7f8f800f0190 msgr2=0x7f8f800f25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 -- 172.21.15.38:0/2253492970 shutdown_connections 2024-02-20T13:17:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.855+0000 7f8f8508d700 1 -- 172.21.15.38:0/2253492970 wait complete. 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.856+0000 7f8f8508d700 1 Processor -- start 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.856+0000 7f8f8508d700 1 -- start start 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.856+0000 7f8f8508d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f80106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.856+0000 7f8f8508d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80106b00 con 0x7f8f800f4690 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f80106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f80106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39552/0 (socket says 172.21.15.38:39552) 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7ffff700 1 -- 172.21.15.38:0/2946036378 learned_addr learned my addr 172.21.15.38:0/2946036378 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7ffff700 1 -- 172.21.15.38:0/2946036378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f70005040 con 0x7f8f800f4690 2024-02-20T13:17:47.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7ffff700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f80106610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8f700089f0 tx=0x7f8f7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f70009320 con 0x7f8f800f4690 2024-02-20T13:17:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8f70013ec0 con 0x7f8f800f4690 2024-02-20T13:17:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f80106d00 con 0x7f8f800f4690 2024-02-20T13:17:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f70014d30 con 0x7f8f800f4690 2024-02-20T13:17:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.857+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f80107190 con 0x7f8f800f4690 2024-02-20T13:17:47.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.858+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8f700095f0 con 0x7f8f800f4690 2024-02-20T13:17:47.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.858+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f80045bb0 con 0x7f8f800f4690 2024-02-20T13:17:47.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.858+0000 7f8f7cff9700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8f68031090 0x7f8f68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:47.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.858+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8f700423e0 con 0x7f8f800f4690 2024-02-20T13:17:47.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.859+0000 7f8f7f7fe700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8f68031090 0x7f8f68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:47.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.859+0000 7f8f7f7fe700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8f68031090 0x7f8f68033550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8f740096a0 tx=0x7f8f74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:47.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:47.863+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8f7001d7f0 con 0x7f8f800f4690 2024-02-20T13:17:48.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.029+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f800f13e0 con 0x7f8f68031090 2024-02-20T13:17:48.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.652+0000 7f8f7cff9700 1 -- 172.21.15.38:0/2946036378 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f8f800f13e0 con 0x7f8f68031090 2024-02-20T13:17:48.654 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi114' with addr '172.21.15.114' 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8f68031090 msgr2=0x7f8f68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8f68031090 0x7f8f68033550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8f740096a0 tx=0x7f8f74006b40).stop 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 msgr2=0x7f8f80106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f80106610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8f700089f0 tx=0x7f8f7000b040).stop 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 shutdown_connections 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8f68031090 0x7f8f68033550 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 --2- 172.21.15.38:0/2946036378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f800f4690 0x7f8f80106610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 >> 172.21.15.38:0/2946036378 conn(0x7f8f800f0190 msgr2=0x7f8f800f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 shutdown_connections 2024-02-20T13:17:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:48.654+0000 7f8f8508d700 1 -- 172.21.15.38:0/2946036378 wait complete. 2024-02-20T13:17:49.314 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:48.031530+0000 mgr.smithi038.powepi (mgr.14156) 7 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:49.314 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:48.136642+0000 mgr.smithi038.powepi (mgr.14156) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:49.314 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:48.652966+0000 mon.smithi038 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:49.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-20T13:17:50.563 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:50 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:48.653322+0000 mgr.smithi038.powepi (mgr.14156) 9 : cephadm [INF] Added host smithi114 2024-02-20T13:17:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 -- 172.21.15.38:0/1945647442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 msgr2=0x7f9d1c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 --2- 172.21.15.38:0/1945647442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c0f58d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9d0c009fb0 tx=0x7f9d0c009b30).stop 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 -- 172.21.15.38:0/1945647442 shutdown_connections 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 --2- 172.21.15.38:0/1945647442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c0f58d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 -- 172.21.15.38:0/1945647442 >> 172.21.15.38:0/1945647442 conn(0x7f9d1c0f0c10 msgr2=0x7f9d1c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 -- 172.21.15.38:0/1945647442 shutdown_connections 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.105+0000 7f9d217ed700 1 -- 172.21.15.38:0/1945647442 wait complete. 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.106+0000 7f9d217ed700 1 Processor -- start 2024-02-20T13:17:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.106+0000 7f9d217ed700 1 -- start start 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.106+0000 7f9d217ed700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.106+0000 7f9d217ed700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c105be0 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d1affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d1affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39572/0 (socket says 172.21.15.38:39572) 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d1affd700 1 -- 172.21.15.38:0/50543740 learned_addr learned my addr 172.21.15.38:0/50543740 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d1affd700 1 -- 172.21.15.38:0/50543740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c005040 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d1affd700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c1056f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f9d0c0089f0 tx=0x7f9d0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0c009320 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9d0c013ec0 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c105de0 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0c014d30 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.107+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c106270 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.108+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9d0c0263a0 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.108+0000 7f9d137fe700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9d080310e0 0x7f9d080335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.108+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9d0c0423a0 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.108+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c045bb0 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.108+0000 7f9d13fff700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9d080310e0 0x7f9d080335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.109+0000 7f9d13fff700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9d080310e0 0x7f9d080335a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f9d040096a0 tx=0x7f9d04006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:51.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.113+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9d0c012920 con 0x7f9d1c0f54b0 2024-02-20T13:17:51.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.277+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d1c0f1de0 con 0x7f9d080310e0 2024-02-20T13:17:51.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.279+0000 7f9d137fe700 1 -- 172.21.15.38:0/50543740 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f9d1c0f1de0 con 0x7f9d080310e0 2024-02-20T13:17:51.280 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-20T13:17:51.280 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9d080310e0 msgr2=0x7f9d080335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9d080310e0 0x7f9d080335a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f9d040096a0 tx=0x7f9d04006b40).stop 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 msgr2=0x7f9d1c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c1056f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f9d0c0089f0 tx=0x7f9d0c00b040).stop 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 shutdown_connections 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9d080310e0 0x7f9d080335a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 --2- 172.21.15.38:0/50543740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d1c0f54b0 0x7f9d1c1056f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 >> 172.21.15.38:0/50543740 conn(0x7f9d1c0f0c10 msgr2=0x7f9d1c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 shutdown_connections 2024-02-20T13:17:51.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:51.280+0000 7f9d217ed700 1 -- 172.21.15.38:0/50543740 wait complete. 2024-02-20T13:17:51.620 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:50.137001+0000 mgr.smithi038.powepi (mgr.14156) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:51.943 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-20T13:17:51.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-20T13:17:52.618 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:51.279795+0000 mgr.smithi038.powepi (mgr.14156) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T13:17:53.317 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:52.137262+0000 mgr.smithi038.powepi (mgr.14156) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:53.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 -- 172.21.15.38:0/1161554287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 msgr2=0x7f1e880f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 --2- 172.21.15.38:0/1161554287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e880f69b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f1e78004760 tx=0x7f1e78009ad0).stop 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 -- 172.21.15.38:0/1161554287 shutdown_connections 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 --2- 172.21.15.38:0/1161554287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e880f69b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 -- 172.21.15.38:0/1161554287 >> 172.21.15.38:0/1161554287 conn(0x7f1e880f1cd0 msgr2=0x7f1e880f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 -- 172.21.15.38:0/1161554287 shutdown_connections 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.810+0000 7f1e8efa5700 1 -- 172.21.15.38:0/1161554287 wait complete. 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8efa5700 1 Processor -- start 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8efa5700 1 -- start start 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8efa5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e881067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8efa5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e88106cb0 con 0x7f1e880f6590 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8cd41700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e881067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8cd41700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e881067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39592/0 (socket says 172.21.15.38:39592) 2024-02-20T13:17:53.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8cd41700 1 -- 172.21.15.38:0/676229208 learned_addr learned my addr 172.21.15.38:0/676229208 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-02-20T13:17:53.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8cd41700 1 -- 172.21.15.38:0/676229208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e78009d70 con 0x7f1e880f6590 2024-02-20T13:17:53.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.811+0000 7f1e8cd41700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e881067c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1e78004170 tx=0x7f1e78009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:53.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.812+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e7800e040 con 0x7f1e880f6590 2024-02-20T13:17:53.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.812+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e88106eb0 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.812+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1e78008ef0 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.812+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e78014c90 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.812+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e88107340 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.813+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1e78014420 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.813+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e88045bb0 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.813+0000 7f1e857fa700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1e74031090 0x7f1e74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.813+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1e78042240 con 0x7f1e880f6590 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.813+0000 7f1e87fff700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1e74031090 0x7f1e74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.813+0000 7f1e87fff700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1e74031090 0x7f1e74033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1e700096a0 tx=0x7f1e70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:53.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.817+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1e78026300 con 0x7f1e880f6590 2024-02-20T13:17:53.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:53.981+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1e881081f0 con 0x7f1e880f6590 2024-02-20T13:17:54.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.214+0000 7f1e857fa700 1 -- 172.21.15.38:0/676229208 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1e78012b20 con 0x7f1e880f6590 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1e74031090 msgr2=0x7f1e74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1e74031090 0x7f1e74033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1e700096a0 tx=0x7f1e70006b40).stop 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 msgr2=0x7f1e881067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e881067c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1e78004170 tx=0x7f1e78009ad0).stop 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 shutdown_connections 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1e74031090 0x7f1e74033550 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 --2- 172.21.15.38:0/676229208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e880f6590 0x7f1e881067c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 >> 172.21.15.38:0/676229208 conn(0x7f1e880f1cd0 msgr2=0x7f1e880f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 shutdown_connections 2024-02-20T13:17:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-02-20T13:17:54.216+0000 7f1e8efa5700 1 -- 172.21.15.38:0/676229208 wait complete. 2024-02-20T13:17:54.219 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-02-20T13:17:54.568 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:54 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:53.983110+0000 mon.smithi038 (mon.0) 102 : audit [INF] from='client.? 172.21.15.38:0/676229208' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-20T13:17:54.840 INFO:tasks.cephadm:Adding mon.smithi038 on smithi038 2024-02-20T13:17:54.840 INFO:tasks.cephadm:Adding mon.smithi114 on smithi114 2024-02-20T13:17:54.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi038:172.21.15.38=smithi038;smithi114:172.21.15.114=smithi114' 2024-02-20T13:17:55.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:54.137742+0000 mgr.smithi038.powepi (mgr.14156) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:55.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:54.215815+0000 mon.smithi038 (mon.0) 103 : audit [INF] from='client.? 172.21.15.38:0/676229208' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-20T13:17:55.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:54.215856+0000 mon.smithi038 (mon.0) 104 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-20T13:17:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 -- 172.21.15.114:0/2449304469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 msgr2=0x7f6ab00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 --2- 172.21.15.114:0/2449304469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab00f58d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f6aa4009fb0 tx=0x7f6aa4009b30).stop 2024-02-20T13:17:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 -- 172.21.15.114:0/2449304469 shutdown_connections 2024-02-20T13:17:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 --2- 172.21.15.114:0/2449304469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab00f58d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 -- 172.21.15.114:0/2449304469 >> 172.21.15.114:0/2449304469 conn(0x7f6ab00f0c10 msgr2=0x7f6ab00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 -- 172.21.15.114:0/2449304469 shutdown_connections 2024-02-20T13:17:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 -- 172.21.15.114:0/2449304469 wait complete. 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.756+0000 7f6ab53b6700 1 Processor -- start 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6ab53b6700 1 -- start start 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6ab53b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab01094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6ab53b6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0060320 con 0x7f6ab00f54b0 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6aaeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab01094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6aaeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab01094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54392/0 (socket says 172.21.15.114:54392) 2024-02-20T13:17:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6aaeffd700 1 -- 172.21.15.114:0/3071027216 learned_addr learned my addr 172.21.15.114:0/3071027216 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.757+0000 7f6aaeffd700 1 -- 172.21.15.114:0/3071027216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa4005040 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6aaeffd700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab01094b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f6aa40089f0 tx=0x7f6aa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aa4009320 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0060520 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6aa4013ec0 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aa4014d30 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0109b60 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.758+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6aa4014420 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.759+0000 7f6a977fe700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6a98031090 0x7f6a98033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.759+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab0045bb0 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.759+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6aa40424c0 con 0x7f6ab00f54b0 2024-02-20T13:17:56.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.759+0000 7f6aae7fc700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6a98031090 0x7f6a98033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:56.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.759+0000 7f6aae7fc700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6a98031090 0x7f6a98033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6aa00096a0 tx=0x7f6aa0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.764+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6aa401d980 con 0x7f6ab00f54b0 2024-02-20T13:17:56.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.921+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 --> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi114:172.21.15.114=smithi114", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ab00f1e70 con 0x7f6a98031090 2024-02-20T13:17:56.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.929+0000 7f6a977fe700 1 -- 172.21.15.114:0/3071027216 <== mgr.14156 v2:172.21.15.38:6800/175894382 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6ab00f1e70 con 0x7f6a98031090 2024-02-20T13:17:56.931 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mon update... 2024-02-20T13:17:56.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6a98031090 msgr2=0x7f6a98033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:56.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6a98031090 0x7f6a98033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6aa00096a0 tx=0x7f6aa0006b40).stop 2024-02-20T13:17:56.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 msgr2=0x7f6ab01094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:56.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab01094b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f6aa40089f0 tx=0x7f6aa400b040).stop 2024-02-20T13:17:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 shutdown_connections 2024-02-20T13:17:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6a98031090 0x7f6a98033550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 --2- 172.21.15.114:0/3071027216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ab00f54b0 0x7f6ab01094b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 >> 172.21.15.114:0/3071027216 conn(0x7f6ab00f0c10 msgr2=0x7f6ab00f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 shutdown_connections 2024-02-20T13:17:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:56.931+0000 7f6ab53b6700 1 -- 172.21.15.114:0/3071027216 wait complete. 2024-02-20T13:17:57.616 DEBUG:teuthology.orchestra.run.smithi114:mon.smithi114> sudo journalctl -f -n 0 -u ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114.service 2024-02-20T13:17:57.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:17:57.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:17:57.646 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:-- Logs begin at Tue 2024-02-20 13:04:46 UTC. -- 2024-02-20T13:17:57.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:56.138096+0000 mgr.smithi038.powepi (mgr.14156) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:17:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:56.930583+0000 mon.smithi038 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:17:58.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:56.924216+0000 mgr.smithi038.powepi (mgr.14156) 15 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi114:172.21.15.114=smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T13:17:58.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:17:56.925711+0000 mgr.smithi038.powepi (mgr.14156) 16 : cephadm [INF] Saving service mon spec with placement smithi038:172.21.15.38=smithi038;smithi114:172.21.15.114=smithi114;count:2 2024-02-20T13:17:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 -- 172.21.15.114:0/1889333320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 msgr2=0x7f71640f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 --2- 172.21.15.114:0/1889333320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f71640f58a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7154009fb0 tx=0x7f7154009b30).stop 2024-02-20T13:17:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 -- 172.21.15.114:0/1889333320 shutdown_connections 2024-02-20T13:17:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 --2- 172.21.15.114:0/1889333320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f71640f58a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 -- 172.21.15.114:0/1889333320 >> 172.21.15.114:0/1889333320 conn(0x7f71640f0be0 msgr2=0x7f71640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 -- 172.21.15.114:0/1889333320 shutdown_connections 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.470+0000 7f71698a8700 1 -- 172.21.15.114:0/1889333320 wait complete. 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f71698a8700 1 Processor -- start 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f71698a8700 1 -- start start 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f71698a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f716410b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f71698a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7164107300 con 0x7f71640f5480 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f7162ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f716410b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f7162ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f716410b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54400/0 (socket says 172.21.15.114:54400) 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.471+0000 7f7162ffd700 1 -- 172.21.15.114:0/3036407240 learned_addr learned my addr 172.21.15.114:0/3036407240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f7162ffd700 1 -- 172.21.15.114:0/3036407240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154005040 con 0x7f71640f5480 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f7162ffd700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f716410b910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f71540089f0 tx=0x7f715400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7154009320 con 0x7f71640f5480 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7164060300 con 0x7f71640f5480 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7154013ec0 con 0x7f71640f5480 2024-02-20T13:17:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7154014d30 con 0x7f71640f5480 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.472+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716410be00 con 0x7f71640f5480 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.473+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7164045bb0 con 0x7f71640f5480 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.473+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f71540095f0 con 0x7f71640f5480 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.473+0000 7f714b7fe700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f714c031090 0x7f714c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.473+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7154042460 con 0x7f71640f5480 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.473+0000 7f71627fc700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f714c031090 0x7f714c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.474+0000 7f71627fc700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f714c031090 0x7f714c033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f71580096a0 tx=0x7f7158006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:17:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.478+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f715401d7f0 con 0x7f71640f5480 2024-02-20T13:17:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.685+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71640faf70 con 0x7f71640f5480 2024-02-20T13:17:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.686+0000 7f714b7fe700 1 -- 172.21.15.114:0/3036407240 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f715401d7f0 con 0x7f71640f5480 2024-02-20T13:17:59.687 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:17:59.687 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f714c031090 msgr2=0x7f714c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f714c031090 0x7f714c033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f71580096a0 tx=0x7f7158006b40).stop 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 msgr2=0x7f716410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f716410b910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f71540089f0 tx=0x7f715400b040).stop 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 shutdown_connections 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f714c031090 0x7f714c033550 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 --2- 172.21.15.114:0/3036407240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71640f5480 0x7f716410b910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 >> 172.21.15.114:0/3036407240 conn(0x7f71640f0be0 msgr2=0x7f71640f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:17:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 shutdown_connections 2024-02-20T13:17:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:17:59.687+0000 7f71698a8700 1 -- 172.21.15.114:0/3036407240 wait complete. 2024-02-20T13:17:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:17:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:17:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:17:58.138344+0000 mgr.smithi038.powepi (mgr.14156) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:00.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:17:59.687166+0000 mon.smithi038 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.114:0/3036407240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:01.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:01.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:01.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:00.138695+0000 mgr.smithi038.powepi (mgr.14156) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:03.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 -- 172.21.15.114:0/516850465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 msgr2=0x7fc8b00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:03.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 --2- 172.21.15.114:0/516850465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b00f69b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc8a4009fb0 tx=0x7fc8a4009b30).stop 2024-02-20T13:18:03.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 -- 172.21.15.114:0/516850465 shutdown_connections 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 --2- 172.21.15.114:0/516850465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b00f69b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 -- 172.21.15.114:0/516850465 >> 172.21.15.114:0/516850465 conn(0x7fc8b00f1cf0 msgr2=0x7fc8b00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 -- 172.21.15.114:0/516850465 shutdown_connections 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.172+0000 7fc8b5838700 1 -- 172.21.15.114:0/516850465 wait complete. 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.173+0000 7fc8b5838700 1 Processor -- start 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.173+0000 7fc8b5838700 1 -- start start 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.173+0000 7fc8b5838700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b0106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.173+0000 7fc8b5838700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b0106cf0 con 0x7fc8b00f6590 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8aeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b0106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8aeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b0106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54410/0 (socket says 172.21.15.114:54410) 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8aeffd700 1 -- 172.21.15.114:0/1847862055 learned_addr learned my addr 172.21.15.114:0/1847862055 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8aeffd700 1 -- 172.21.15.114:0/1847862055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4005040 con 0x7fc8b00f6590 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8aeffd700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b0106800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc8a40089f0 tx=0x7fc8a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8a4009320 con 0x7fc8b00f6590 2024-02-20T13:18:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc8a4013ec0 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b0106ef0 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.174+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8a4014d30 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.175+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b0107380 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.176+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc8a40095f0 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.175+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b0045bb0 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.176+0000 7fc8977fe700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc8980310e0 0x7fc8980335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.176+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc8a40423a0 con 0x7fc8b00f6590 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.176+0000 7fc8ae7fc700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc8980310e0 0x7fc8980335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.176+0000 7fc8ae7fc700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc8980310e0 0x7fc8980335a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc8a00096a0 tx=0x7fc8a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.181+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc8a401d7f0 con 0x7fc8b00f6590 2024-02-20T13:18:03.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.389+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8b00fc080 con 0x7fc8b00f6590 2024-02-20T13:18:03.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.390+0000 7fc8977fe700 1 -- 172.21.15.114:0/1847862055 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8a400fe40 con 0x7fc8b00f6590 2024-02-20T13:18:03.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:03.392 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc8980310e0 msgr2=0x7fc8980335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc8980310e0 0x7fc8980335a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc8a00096a0 tx=0x7fc8a0006b40).stop 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 msgr2=0x7fc8b0106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b0106800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc8a40089f0 tx=0x7fc8a400b040).stop 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 shutdown_connections 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc8980310e0 0x7fc8980335a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 --2- 172.21.15.114:0/1847862055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b00f6590 0x7fc8b0106800 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 >> 172.21.15.114:0/1847862055 conn(0x7fc8b00f1cf0 msgr2=0x7fc8b00f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 shutdown_connections 2024-02-20T13:18:03.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:03.391+0000 7fc8b5838700 1 -- 172.21.15.114:0/1847862055 wait complete. 2024-02-20T13:18:03.394 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:02.139049+0000 mgr.smithi038.powepi (mgr.14156) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:04.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:03.391382+0000 mon.smithi038 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.114:0/1847862055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:04.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:04.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:05.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:04.139260+0000 mgr.smithi038.powepi (mgr.14156) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 -- 172.21.15.114:0/3580098016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 msgr2=0x7f6fe80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 --2- 172.21.15.114:0/3580098016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe80f69b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6fdc004760 tx=0x7f6fdc009ad0).stop 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 -- 172.21.15.114:0/3580098016 shutdown_connections 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 --2- 172.21.15.114:0/3580098016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe80f69b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 -- 172.21.15.114:0/3580098016 >> 172.21.15.114:0/3580098016 conn(0x7f6fe80f1cd0 msgr2=0x7f6fe80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 -- 172.21.15.114:0/3580098016 shutdown_connections 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.784+0000 7f6fee445700 1 -- 172.21.15.114:0/3580098016 wait complete. 2024-02-20T13:18:06.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fee445700 1 Processor -- start 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fee445700 1 -- start start 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fee445700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fee445700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe8106cb0 con 0x7f6fe80f6590 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fe7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fe7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:42118/0 (socket says 172.21.15.114:42118) 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.785+0000 7f6fe7fff700 1 -- 172.21.15.114:0/3441693142 learned_addr learned my addr 172.21.15.114:0/3441693142 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:06.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fe7fff700 1 -- 172.21.15.114:0/3441693142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc009d70 con 0x7f6fe80f6590 2024-02-20T13:18:06.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fe7fff700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe81067c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6fdc004170 tx=0x7f6fdc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fdc00e040 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe8106eb0 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6fdc008ef0 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fdc014c90 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.786+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe8107340 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.787+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6fdc0283f0 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.787+0000 7f6fe4ff9700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6fd0031090 0x7f6fd0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.787+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6fdc042240 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.787+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe80591e0 con 0x7f6fe80f6590 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.787+0000 7f6fe77fe700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6fd0031090 0x7f6fd0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:06.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.788+0000 7f6fe77fe700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6fd0031090 0x7f6fd0033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f6fd80096a0 tx=0x7f6fd8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:06.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:06.792+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6fdc0268f0 con 0x7f6fe80f6590 2024-02-20T13:18:07.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.001+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fe8045bb0 con 0x7f6fe80f6590 2024-02-20T13:18:07.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.001+0000 7f6fe4ff9700 1 -- 172.21.15.114:0/3441693142 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fdc026300 con 0x7f6fe80f6590 2024-02-20T13:18:07.003 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:07.003 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6fd0031090 msgr2=0x7f6fd0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6fd0031090 0x7f6fd0033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f6fd80096a0 tx=0x7f6fd8006b40).stop 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 msgr2=0x7f6fe81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe81067c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6fdc004170 tx=0x7f6fdc009ad0).stop 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 shutdown_connections 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6fd0031090 0x7f6fd0033550 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 --2- 172.21.15.114:0/3441693142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fe80f6590 0x7f6fe81067c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:07.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 >> 172.21.15.114:0/3441693142 conn(0x7f6fe80f1cd0 msgr2=0x7f6fe80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:07.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 shutdown_connections 2024-02-20T13:18:07.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:07.003+0000 7f6fee445700 1 -- 172.21.15.114:0/3441693142 wait complete. 2024-02-20T13:18:07.006 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:07.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:06.139634+0000 mgr.smithi038.powepi (mgr.14156) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:07.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:07.003145+0000 mon.smithi038 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.114:0/3441693142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:08.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:08.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:09.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:08.139915+0000 mgr.smithi038.powepi (mgr.14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:10.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1573007886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78060320 msgr2=0x7fdb780f91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 --2- 172.21.15.114:0/1573007886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78060320 0x7fdb780f91a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdb6c009fb0 tx=0x7fdb6c009b30).stop 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1573007886 shutdown_connections 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 --2- 172.21.15.114:0/1573007886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78060320 0x7fdb780f91a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1573007886 >> 172.21.15.114:0/1573007886 conn(0x7fdb780f0be0 msgr2=0x7fdb780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1573007886 shutdown_connections 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1573007886 wait complete. 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.581+0000 7fdb7ec55700 1 Processor -- start 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.582+0000 7fdb7ec55700 1 -- start start 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.582+0000 7fdb7ec55700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 0x7fdb780f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.582+0000 7fdb7ec55700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780f7000 con 0x7fdb780f6be0 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.582+0000 7fdb7c9f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 0x7fdb780f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.582+0000 7fdb7c9f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 0x7fdb780f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:42146/0 (socket says 172.21.15.114:42146) 2024-02-20T13:18:10.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.582+0000 7fdb7c9f1700 1 -- 172.21.15.114:0/1585554806 learned_addr learned my addr 172.21.15.114:0/1585554806 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb7c9f1700 1 -- 172.21.15.114:0/1585554806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c005040 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb7c9f1700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 0x7fdb780f50a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdb6c0134d0 tx=0x7fdb6c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c013ee0 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb780f5590 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdb6c013680 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c014420 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.583+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb780f5a80 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.584+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdb6c025060 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.584+0000 7fdb757fa700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb68031090 0x7fdb68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.584+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb780591e0 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.584+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdb6c0422a0 con 0x7fdb780f6be0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.584+0000 7fdb77fff700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb68031090 0x7fdb68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.584+0000 7fdb77fff700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb68031090 0x7fdb68033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fdb640096a0 tx=0x7fdb64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:10.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.589+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdb6c0095f0 con 0x7fdb780f6be0 2024-02-20T13:18:10.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.798+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb780f5e40 con 0x7fdb780f6be0 2024-02-20T13:18:10.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.799+0000 7fdb757fa700 1 -- 172.21.15.114:0/1585554806 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb6c004490 con 0x7fdb780f6be0 2024-02-20T13:18:10.799 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:10.800 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb68031090 msgr2=0x7fdb68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb68031090 0x7fdb68033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fdb640096a0 tx=0x7fdb64006b40).stop 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 msgr2=0x7fdb780f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 0x7fdb780f50a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdb6c0134d0 tx=0x7fdb6c009b30).stop 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 shutdown_connections 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb68031090 0x7fdb68033550 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 --2- 172.21.15.114:0/1585554806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb780f6be0 0x7fdb780f50a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 >> 172.21.15.114:0/1585554806 conn(0x7fdb780f0be0 msgr2=0x7fdb780f17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 shutdown_connections 2024-02-20T13:18:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:10.800+0000 7fdb7ec55700 1 -- 172.21.15.114:0/1585554806 wait complete. 2024-02-20T13:18:10.802 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:11.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:10.140171+0000 mgr.smithi038.powepi (mgr.14156) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:11.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:10.799271+0000 mon.smithi038 (mon.0) 109 : audit [DBG] from='client.? 172.21.15.114:0/1585554806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:12.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:12.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:13.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:12.140509+0000 mgr.smithi038.powepi (mgr.14156) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.188+0000 7f3935c5a700 1 -- 172.21.15.114:0/3127360439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 msgr2=0x7f39300f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.188+0000 7f3935c5a700 1 --2- 172.21.15.114:0/3127360439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39300f9050 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3924009fb0 tx=0x7f3924009b30).stop 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 -- 172.21.15.114:0/3127360439 shutdown_connections 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 --2- 172.21.15.114:0/3127360439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39300f9050 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 -- 172.21.15.114:0/3127360439 >> 172.21.15.114:0/3127360439 conn(0x7f39300f0a30 msgr2=0x7f39300f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 -- 172.21.15.114:0/3127360439 shutdown_connections 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 -- 172.21.15.114:0/3127360439 wait complete. 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 Processor -- start 2024-02-20T13:18:14.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.189+0000 7f3935c5a700 1 -- start start 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.190+0000 7f3935c5a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39301065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.190+0000 7f3935c5a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3930106aa0 con 0x7f39300602c0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.190+0000 7f3934c58700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39301065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.190+0000 7f3934c58700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39301065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:42164/0 (socket says 172.21.15.114:42164) 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.190+0000 7f3934c58700 1 -- 172.21.15.114:0/3954286853 learned_addr learned my addr 172.21.15.114:0/3954286853 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.190+0000 7f3934c58700 1 -- 172.21.15.114:0/3954286853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3924005040 con 0x7f39300602c0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.191+0000 7f3934c58700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39301065b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3924013fd0 tx=0x7f3924009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.191+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3924018070 con 0x7f39300602c0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.191+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3930106ca0 con 0x7f39300602c0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.191+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3924007b10 con 0x7f39300602c0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.191+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f392401c3c0 con 0x7f39300602c0 2024-02-20T13:18:14.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.191+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3930107130 con 0x7f39300602c0 2024-02-20T13:18:14.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.192+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3930045bb0 con 0x7f39300602c0 2024-02-20T13:18:14.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.193+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3924007c80 con 0x7f39300602c0 2024-02-20T13:18:14.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.193+0000 7f392d7fa700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3920031090 0x7f3920033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:14.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.193+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3924042f80 con 0x7f39300602c0 2024-02-20T13:18:14.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.193+0000 7f392ffff700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3920031090 0x7f3920033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:14.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.194+0000 7f392ffff700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3920031090 0x7f3920033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f391c0096a0 tx=0x7f391c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:14.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.197+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3924025b40 con 0x7f39300602c0 2024-02-20T13:18:14.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.405+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3930107fe0 con 0x7f39300602c0 2024-02-20T13:18:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.405+0000 7f392d7fa700 1 -- 172.21.15.114:0/3954286853 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3924025b40 con 0x7f39300602c0 2024-02-20T13:18:14.406 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:14.406 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3920031090 msgr2=0x7f3920033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3920031090 0x7f3920033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f391c0096a0 tx=0x7f391c006b40).stop 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 msgr2=0x7f39301065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39301065b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3924013fd0 tx=0x7f3924009b30).stop 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 shutdown_connections 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3920031090 0x7f3920033550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 --2- 172.21.15.114:0/3954286853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39300602c0 0x7f39301065b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 >> 172.21.15.114:0/3954286853 conn(0x7f39300f0a30 msgr2=0x7f39300f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 shutdown_connections 2024-02-20T13:18:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:14.407+0000 7f3935c5a700 1 -- 172.21.15.114:0/3954286853 wait complete. 2024-02-20T13:18:14.409 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:15.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:14.140785+0000 mgr.smithi038.powepi (mgr.14156) 25 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:15.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:14.406120+0000 mon.smithi038 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.114:0/3954286853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:16.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:16.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:17.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:16.141114+0000 mgr.smithi038.powepi (mgr.14156) 26 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:17.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3113883746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 msgr2=0x7f9c640f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:17.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 --2- 172.21.15.114:0/3113883746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c640f69b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9c54004760 tx=0x7f9c54009ad0).stop 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3113883746 shutdown_connections 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 --2- 172.21.15.114:0/3113883746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c640f69b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3113883746 >> 172.21.15.114:0/3113883746 conn(0x7f9c640f1cd0 msgr2=0x7f9c640f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3113883746 shutdown_connections 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.891+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3113883746 wait complete. 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6c76f700 1 Processor -- start 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6c76f700 1 -- start start 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6c76f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c6406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:17.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6c76f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64069f00 con 0x7f9c640f6590 2024-02-20T13:18:17.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6a50b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c6406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6a50b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c6406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50682/0 (socket says 172.21.15.114:50682) 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.892+0000 7f9c6a50b700 1 -- 172.21.15.114:0/3951643029 learned_addr learned my addr 172.21.15.114:0/3951643029 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c6a50b700 1 -- 172.21.15.114:0/3951643029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54009d70 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c6a50b700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c6406ba80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9c54004170 tx=0x7f9c54009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c5400e040 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6406a0a0 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9c54008ef0 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c54014c90 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.893+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6406a530 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.894+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9c540283f0 con 0x7f9c640f6590 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.894+0000 7f9c5affd700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c50031090 0x7f9c50033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.894+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9c54042220 con 0x7f9c640f6590 2024-02-20T13:18:17.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.894+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c640591e0 con 0x7f9c640f6590 2024-02-20T13:18:17.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.895+0000 7f9c69d0a700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c50031090 0x7f9c50033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:17.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.895+0000 7f9c69d0a700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c50031090 0x7f9c50033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9c600096a0 tx=0x7f9c60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:17.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:17.899+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9c540268f0 con 0x7f9c640f6590 2024-02-20T13:18:18.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.103+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c6406ab90 con 0x7f9c640f6590 2024-02-20T13:18:18.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.104+0000 7f9c5affd700 1 -- 172.21.15.114:0/3951643029 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c54026300 con 0x7f9c640f6590 2024-02-20T13:18:18.104 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:18.104 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:18.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c50031090 msgr2=0x7f9c50033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:18.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c50031090 0x7f9c50033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9c600096a0 tx=0x7f9c60006b40).stop 2024-02-20T13:18:18.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 msgr2=0x7f9c6406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:18.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c6406ba80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9c54004170 tx=0x7f9c54009ad0).stop 2024-02-20T13:18:18.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 shutdown_connections 2024-02-20T13:18:18.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c50031090 0x7f9c50033550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:18.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 --2- 172.21.15.114:0/3951643029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c640f6590 0x7f9c6406ba80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:18.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 >> 172.21.15.114:0/3951643029 conn(0x7f9c640f1cd0 msgr2=0x7f9c640f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:18.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 shutdown_connections 2024-02-20T13:18:18.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:18.105+0000 7f9c6c76f700 1 -- 172.21.15.114:0/3951643029 wait complete. 2024-02-20T13:18:18.107 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:18.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:18.104274+0000 mon.smithi038 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.114:0/3951643029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:19.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:18.141417+0000 mgr.smithi038.powepi (mgr.14156) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:19.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:19.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 -- 172.21.15.114:0/376359967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 msgr2=0x7fc8740f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 --2- 172.21.15.114:0/376359967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8740f4c70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc864009fb0 tx=0x7fc864009b30).stop 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 -- 172.21.15.114:0/376359967 shutdown_connections 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 --2- 172.21.15.114:0/376359967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8740f4c70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 -- 172.21.15.114:0/376359967 >> 172.21.15.114:0/376359967 conn(0x7fc8740f0390 msgr2=0x7fc8740f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 -- 172.21.15.114:0/376359967 shutdown_connections 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.615+0000 7fc87a74f700 1 -- 172.21.15.114:0/376359967 wait complete. 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc87a74f700 1 Processor -- start 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc87a74f700 1 -- start start 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc87a74f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8741067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc87a74f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874106ca0 con 0x7fc8740f4850 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc873fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8741067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc873fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8741067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50690/0 (socket says 172.21.15.114:50690) 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.616+0000 7fc873fff700 1 -- 172.21.15.114:0/699863147 learned_addr learned my addr 172.21.15.114:0/699863147 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc873fff700 1 -- 172.21.15.114:0/699863147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864005040 con 0x7fc8740f4850 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc873fff700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8741067b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fc8640089f0 tx=0x7fc86400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc864009320 con 0x7fc8740f4850 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc874106ea0 con 0x7fc8740f4850 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc864013ec0 con 0x7fc8740f4850 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc864014d30 con 0x7fc8740f4850 2024-02-20T13:18:21.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.617+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc874107330 con 0x7fc8740f4850 2024-02-20T13:18:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.618+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc8640095f0 con 0x7fc8740f4850 2024-02-20T13:18:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.618+0000 7fc870ff9700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc85c031090 0x7fc85c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.618+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc874045bb0 con 0x7fc8740f4850 2024-02-20T13:18:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.618+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc8640424a0 con 0x7fc8740f4850 2024-02-20T13:18:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.618+0000 7fc8737fe700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc85c031090 0x7fc85c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.619+0000 7fc8737fe700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc85c031090 0x7fc85c033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fc8680096a0 tx=0x7fc868006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:21.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.623+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc86401d7f0 con 0x7fc8740f4850 2024-02-20T13:18:21.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:20.141775+0000 mgr.smithi038.powepi (mgr.14156) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:21.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.828+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc874108130 con 0x7fc8740f4850 2024-02-20T13:18:21.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.829+0000 7fc870ff9700 1 -- 172.21.15.114:0/699863147 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc86401d7f0 con 0x7fc8740f4850 2024-02-20T13:18:21.830 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:21.830 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:21.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc85c031090 msgr2=0x7fc85c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:21.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc85c031090 0x7fc85c033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fc8680096a0 tx=0x7fc868006b40).stop 2024-02-20T13:18:21.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 msgr2=0x7fc8741067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8741067b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fc8640089f0 tx=0x7fc86400b040).stop 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 shutdown_connections 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc85c031090 0x7fc85c033550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 --2- 172.21.15.114:0/699863147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8740f4850 0x7fc8741067b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 >> 172.21.15.114:0/699863147 conn(0x7fc8740f0390 msgr2=0x7fc8740f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 shutdown_connections 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:21.831+0000 7fc87a74f700 1 -- 172.21.15.114:0/699863147 wait complete. 2024-02-20T13:18:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:22.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:21.829860+0000 mon.smithi038 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.114:0/699863147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:23.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:23.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:22.142153+0000 mgr.smithi038.powepi (mgr.14156) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:25.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 -- 172.21.15.114:0/897181572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 msgr2=0x7f13200f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 --2- 172.21.15.114:0/897181572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f13200f56f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1310009fb0 tx=0x7f1310009b30).stop 2024-02-20T13:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 -- 172.21.15.114:0/897181572 shutdown_connections 2024-02-20T13:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 --2- 172.21.15.114:0/897181572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f13200f56f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 -- 172.21.15.114:0/897181572 >> 172.21.15.114:0/897181572 conn(0x7f13200f0a30 msgr2=0x7f13200f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 -- 172.21.15.114:0/897181572 shutdown_connections 2024-02-20T13:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.330+0000 7f132535f700 1 -- 172.21.15.114:0/897181572 wait complete. 2024-02-20T13:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f132535f700 1 Processor -- start 2024-02-20T13:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f132535f700 1 -- start start 2024-02-20T13:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f132535f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f1320105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f132535f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1320105a00 con 0x7f13200f52d0 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f131ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f1320105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f131ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f1320105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55630/0 (socket says 172.21.15.114:55630) 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.331+0000 7f131ffff700 1 -- 172.21.15.114:0/2100364648 learned_addr learned my addr 172.21.15.114:0/2100364648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f131ffff700 1 -- 172.21.15.114:0/2100364648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310005040 con 0x7f13200f52d0 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f131ffff700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f1320105510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f13100089f0 tx=0x7f131000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1310009320 con 0x7f13200f52d0 2024-02-20T13:18:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1320105c00 con 0x7f13200f52d0 2024-02-20T13:18:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1310013ec0 con 0x7f13200f52d0 2024-02-20T13:18:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1310014d30 con 0x7f13200f52d0 2024-02-20T13:18:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.332+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1320106090 con 0x7f13200f52d0 2024-02-20T13:18:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.333+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f13100095f0 con 0x7f13200f52d0 2024-02-20T13:18:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.333+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1320045bb0 con 0x7f13200f52d0 2024-02-20T13:18:25.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.333+0000 7f131cff9700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1308031090 0x7f1308033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:25.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.333+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13100423e0 con 0x7f13200f52d0 2024-02-20T13:18:25.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.333+0000 7f131f7fe700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1308031090 0x7f1308033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:25.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.333+0000 7f131f7fe700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1308031090 0x7f1308033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f13140096a0 tx=0x7f1314006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:25.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.338+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f131001d980 con 0x7f13200f52d0 2024-02-20T13:18:25.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.545+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1320106f40 con 0x7f13200f52d0 2024-02-20T13:18:25.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.545+0000 7f131cff9700 1 -- 172.21.15.114:0/2100364648 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f131001d380 con 0x7f13200f52d0 2024-02-20T13:18:25.546 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:25.546 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:25.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1308031090 msgr2=0x7f1308033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:25.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1308031090 0x7f1308033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f13140096a0 tx=0x7f1314006b40).stop 2024-02-20T13:18:25.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 msgr2=0x7f1320105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:25.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f1320105510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f13100089f0 tx=0x7f131000b040).stop 2024-02-20T13:18:25.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 shutdown_connections 2024-02-20T13:18:25.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1308031090 0x7f1308033550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:25.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 --2- 172.21.15.114:0/2100364648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13200f52d0 0x7f1320105510 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:25.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 >> 172.21.15.114:0/2100364648 conn(0x7f13200f0a30 msgr2=0x7f13200f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:25.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 shutdown_connections 2024-02-20T13:18:25.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:25.547+0000 7f132535f700 1 -- 172.21.15.114:0/2100364648 wait complete. 2024-02-20T13:18:25.549 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:24.142498+0000 mgr.smithi038.powepi (mgr.14156) 30 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:26.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:26 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:25.545991+0000 mon.smithi038 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.114:0/2100364648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:27.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:27.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:26.142909+0000 mgr.smithi038.powepi (mgr.14156) 31 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 -- 172.21.15.114:0/2471213424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 msgr2=0x7f47f80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 --2- 172.21.15.114:0/2471213424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f80f69b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f47ec004760 tx=0x7f47ec009ad0).stop 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 -- 172.21.15.114:0/2471213424 shutdown_connections 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 --2- 172.21.15.114:0/2471213424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f80f69b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 -- 172.21.15.114:0/2471213424 >> 172.21.15.114:0/2471213424 conn(0x7f47f80f1cd0 msgr2=0x7f47f80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 -- 172.21.15.114:0/2471213424 shutdown_connections 2024-02-20T13:18:28.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 -- 172.21.15.114:0/2471213424 wait complete. 2024-02-20T13:18:28.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 Processor -- start 2024-02-20T13:18:28.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.990+0000 7f47fd6a4700 1 -- start start 2024-02-20T13:18:28.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47fd6a4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:28.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47fd6a4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8106cb0 con 0x7f47f80f6590 2024-02-20T13:18:28.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47f6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:28.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47f6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55650/0 (socket says 172.21.15.114:55650) 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47f6ffd700 1 -- 172.21.15.114:0/257980274 learned_addr learned my addr 172.21.15.114:0/257980274 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47f6ffd700 1 -- 172.21.15.114:0/257980274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec009d70 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47f6ffd700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f81067c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f47ec004170 tx=0x7f47ec009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.991+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ec00e040 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.992+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8106eb0 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.992+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f47ec008ef0 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.992+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ec014c90 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.992+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f8107340 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.992+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f47ec014420 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.992+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f8045bb0 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.993+0000 7f47df7fe700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f47e0031090 0x7f47e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.993+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47ec042300 con 0x7f47f80f6590 2024-02-20T13:18:28.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.993+0000 7f47f67fc700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f47e0031090 0x7f47e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:28.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.993+0000 7f47f67fc700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f47e0031090 0x7f47e0033550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f47e80096a0 tx=0x7f47e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:28.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:28.997+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f47ec026300 con 0x7f47f80f6590 2024-02-20T13:18:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.205+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47f81081f0 con 0x7f47f80f6590 2024-02-20T13:18:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.205+0000 7f47df7fe700 1 -- 172.21.15.114:0/257980274 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47ec012b20 con 0x7f47f80f6590 2024-02-20T13:18:29.206 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:29.206 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f47e0031090 msgr2=0x7f47e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f47e0031090 0x7f47e0033550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f47e80096a0 tx=0x7f47e8006b40).stop 2024-02-20T13:18:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 msgr2=0x7f47f81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f81067c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f47ec004170 tx=0x7f47ec009ad0).stop 2024-02-20T13:18:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 shutdown_connections 2024-02-20T13:18:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f47e0031090 0x7f47e0033550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 --2- 172.21.15.114:0/257980274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47f80f6590 0x7f47f81067c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 >> 172.21.15.114:0/257980274 conn(0x7f47f80f1cd0 msgr2=0x7f47f80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 shutdown_connections 2024-02-20T13:18:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:29.207+0000 7f47fd6a4700 1 -- 172.21.15.114:0/257980274 wait complete. 2024-02-20T13:18:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:28.143243+0000 mgr.smithi038.powepi (mgr.14156) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:28.274263+0000 mon.smithi038 (mon.0) 114 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:18:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:28.275335+0000 mon.smithi038 (mon.0) 115 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:18:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:29.206176+0000 mon.smithi038 (mon.0) 116 : audit [DBG] from='client.? 172.21.15.114:0/257980274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:30.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:30.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:30.143623+0000 mgr.smithi038.powepi (mgr.14156) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/988849183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 msgr2=0x7fc9c00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 --2- 172.21.15.114:0/988849183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c00f69b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc9b4004760 tx=0x7fc9b4009ad0).stop 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/988849183 shutdown_connections 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 --2- 172.21.15.114:0/988849183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c00f69b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/988849183 >> 172.21.15.114:0/988849183 conn(0x7fc9c00f1cd0 msgr2=0x7fc9c00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/988849183 shutdown_connections 2024-02-20T13:18:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.701+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/988849183 wait complete. 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9c5b4f700 1 Processor -- start 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9c5b4f700 1 -- start start 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9c5b4f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9c5b4f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c0106cb0 con 0x7fc9c00f6590 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55678/0 (socket says 172.21.15.114:55678) 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.702+0000 7fc9bf7fe700 1 -- 172.21.15.114:0/4272157676 learned_addr learned my addr 172.21.15.114:0/4272157676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9bf7fe700 1 -- 172.21.15.114:0/4272157676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4009d70 con 0x7fc9c00f6590 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9bf7fe700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c01067c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc9b4004170 tx=0x7fc9b4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b400e040 con 0x7fc9c00f6590 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c0106eb0 con 0x7fc9c00f6590 2024-02-20T13:18:32.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc9b4008ef0 con 0x7fc9c00f6590 2024-02-20T13:18:32.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b4014c90 con 0x7fc9c00f6590 2024-02-20T13:18:32.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.703+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c0107340 con 0x7fc9c00f6590 2024-02-20T13:18:32.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.704+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc9b40283f0 con 0x7fc9c00f6590 2024-02-20T13:18:32.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.704+0000 7fc9a7fff700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9a8031090 0x7fc9a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:32.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.704+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc9b4042300 con 0x7fc9c00f6590 2024-02-20T13:18:32.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.704+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c0045bb0 con 0x7fc9c00f6590 2024-02-20T13:18:32.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.704+0000 7fc9beffd700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9a8031090 0x7fc9a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:32.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.705+0000 7fc9beffd700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9a8031090 0x7fc9a8033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fc9b00096a0 tx=0x7fc9b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:32.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.709+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc9b40268f0 con 0x7fc9c00f6590 2024-02-20T13:18:32.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.914+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9c01080c0 con 0x7fc9c00f6590 2024-02-20T13:18:32.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.915+0000 7fc9a7fff700 1 -- 172.21.15.114:0/4272157676 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc9b4026300 con 0x7fc9c00f6590 2024-02-20T13:18:32.917 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:32.917 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:32.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9a8031090 msgr2=0x7fc9a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9a8031090 0x7fc9a8033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fc9b00096a0 tx=0x7fc9b0006b40).stop 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 msgr2=0x7fc9c01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c01067c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc9b4004170 tx=0x7fc9b4009ad0).stop 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 shutdown_connections 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9a8031090 0x7fc9a8033550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 --2- 172.21.15.114:0/4272157676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f6590 0x7fc9c01067c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 >> 172.21.15.114:0/4272157676 conn(0x7fc9c00f1cd0 msgr2=0x7fc9c00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 shutdown_connections 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:32.917+0000 7fc9c5b4f700 1 -- 172.21.15.114:0/4272157676 wait complete. 2024-02-20T13:18:32.919 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:32.143905+0000 mgr.smithi038.powepi (mgr.14156) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:32.916666+0000 mon.smithi038 (mon.0) 117 : audit [DBG] from='client.? 172.21.15.114:0/4272157676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:34.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:34.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:34.144128+0000 mgr.smithi038.powepi (mgr.14156) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 -- 172.21.15.114:0/2461708518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 msgr2=0x7f34d80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 --2- 172.21.15.114:0/2461708518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d80f69b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f34c8004760 tx=0x7f34c8009ad0).stop 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 -- 172.21.15.114:0/2461708518 shutdown_connections 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 --2- 172.21.15.114:0/2461708518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d80f69b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 -- 172.21.15.114:0/2461708518 >> 172.21.15.114:0/2461708518 conn(0x7f34d80f1cd0 msgr2=0x7f34d80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 -- 172.21.15.114:0/2461708518 shutdown_connections 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.451+0000 7f34dfdf9700 1 -- 172.21.15.114:0/2461708518 wait complete. 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34dfdf9700 1 Processor -- start 2024-02-20T13:18:36.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34dfdf9700 1 -- start start 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34dfdf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34dfdf9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8069f00 con 0x7f34d80f6590 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34ddb95700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34ddb95700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54864/0 (socket says 172.21.15.114:54864) 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.452+0000 7f34ddb95700 1 -- 172.21.15.114:0/676829396 learned_addr learned my addr 172.21.15.114:0/676829396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34ddb95700 1 -- 172.21.15.114:0/676829396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009d70 con 0x7f34d80f6590 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34ddb95700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d806ba80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f34c8004170 tx=0x7f34c8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c800e040 con 0x7f34d80f6590 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f34c8008ef0 con 0x7f34d80f6590 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d806a0a0 con 0x7f34d80f6590 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c8014c90 con 0x7f34d80f6590 2024-02-20T13:18:36.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.453+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d806a530 con 0x7f34d80f6590 2024-02-20T13:18:36.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.454+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f34c8014420 con 0x7f34d80f6590 2024-02-20T13:18:36.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.454+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d80591e0 con 0x7f34d80f6590 2024-02-20T13:18:36.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.454+0000 7f34ce7fc700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34c4031090 0x7f34c4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:36.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.454+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f34c8042320 con 0x7f34d80f6590 2024-02-20T13:18:36.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.454+0000 7f34dd394700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34c4031090 0x7f34c4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:36.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.455+0000 7f34dd394700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34c4031090 0x7f34c4033550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f34d40096a0 tx=0x7f34d4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:36.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.459+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f34c8026300 con 0x7f34d80f6590 2024-02-20T13:18:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.665+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34d80fc080 con 0x7f34d80f6590 2024-02-20T13:18:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.666+0000 7f34ce7fc700 1 -- 172.21.15.114:0/676829396 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34c8012b20 con 0x7f34d80f6590 2024-02-20T13:18:36.668 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:36.668 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34c4031090 msgr2=0x7f34c4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34c4031090 0x7f34c4033550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f34d40096a0 tx=0x7f34d4006b40).stop 2024-02-20T13:18:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 msgr2=0x7f34d806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d806ba80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f34c8004170 tx=0x7f34c8009ad0).stop 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 shutdown_connections 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34c4031090 0x7f34c4033550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 --2- 172.21.15.114:0/676829396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34d80f6590 0x7f34d806ba80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 >> 172.21.15.114:0/676829396 conn(0x7f34d80f1cd0 msgr2=0x7f34d80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 shutdown_connections 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:36.668+0000 7f34dfdf9700 1 -- 172.21.15.114:0/676829396 wait complete. 2024-02-20T13:18:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:37.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:36.144473+0000 mgr.smithi038.powepi (mgr.14156) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:36.667869+0000 mon.smithi038 (mon.0) 118 : audit [DBG] from='client.? 172.21.15.114:0/676829396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:38.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:38.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:39.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:38.144799+0000 mgr.smithi038.powepi (mgr.14156) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 -- 172.21.15.114:0/3676780652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 msgr2=0x7f73800f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 --2- 172.21.15.114:0/3676780652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f73800f69b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7370004760 tx=0x7f7370009ad0).stop 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 -- 172.21.15.114:0/3676780652 shutdown_connections 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 --2- 172.21.15.114:0/3676780652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f73800f69b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 -- 172.21.15.114:0/3676780652 >> 172.21.15.114:0/3676780652 conn(0x7f73800f1cd0 msgr2=0x7f73800f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 -- 172.21.15.114:0/3676780652 shutdown_connections 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.160+0000 7f7384dd4700 1 -- 172.21.15.114:0/3676780652 wait complete. 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f7384dd4700 1 Processor -- start 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f7384dd4700 1 -- start start 2024-02-20T13:18:40.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f7384dd4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f738006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f7384dd4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7380069f00 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f737e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f738006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f737e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f738006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54880/0 (socket says 172.21.15.114:54880) 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.161+0000 7f737e59c700 1 -- 172.21.15.114:0/1944948219 learned_addr learned my addr 172.21.15.114:0/1944948219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f737e59c700 1 -- 172.21.15.114:0/1944948219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370009d70 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f737e59c700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f738006ba80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7370004170 tx=0x7f7370009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f737000e040 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7370008ef0 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7370014c90 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738006a0a0 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.162+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738006a530 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.163+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7370014420 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.163+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73800591e0 con 0x7f73800f6590 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.163+0000 7f736effd700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7368031130 0x7f73680335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:40.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.163+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f73700422b0 con 0x7f73800f6590 2024-02-20T13:18:40.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.163+0000 7f737dd9b700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7368031130 0x7f73680335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:40.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.164+0000 7f737dd9b700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7368031130 0x7f73680335f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f73740096a0 tx=0x7f7374006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:40.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.168+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7370026300 con 0x7f73800f6590 2024-02-20T13:18:40.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.374+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73800fc080 con 0x7f73800f6590 2024-02-20T13:18:40.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.375+0000 7f736effd700 1 -- 172.21.15.114:0/1944948219 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7370012b20 con 0x7f73800f6590 2024-02-20T13:18:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:40.376 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7368031130 msgr2=0x7f73680335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7368031130 0x7f73680335f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f73740096a0 tx=0x7f7374006b40).stop 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 msgr2=0x7f738006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f738006ba80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7370004170 tx=0x7f7370009ad0).stop 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 shutdown_connections 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7368031130 0x7f73680335f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 --2- 172.21.15.114:0/1944948219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73800f6590 0x7f738006ba80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 >> 172.21.15.114:0/1944948219 conn(0x7f73800f1cd0 msgr2=0x7f73800f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 shutdown_connections 2024-02-20T13:18:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:40.376+0000 7f7384dd4700 1 -- 172.21.15.114:0/1944948219 wait complete. 2024-02-20T13:18:40.379 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:40.145139+0000 mgr.smithi038.powepi (mgr.14156) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:40.376338+0000 mon.smithi038 (mon.0) 119 : audit [DBG] from='client.? 172.21.15.114:0/1944948219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:41.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:41.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:43.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:42.145497+0000 mgr.smithi038.powepi (mgr.14156) 39 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 -- 172.21.15.114:0/2403125040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 msgr2=0x7f4b240f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 --2- 172.21.15.114:0/2403125040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b240f69b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4b14004760 tx=0x7f4b14009ad0).stop 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 -- 172.21.15.114:0/2403125040 shutdown_connections 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 --2- 172.21.15.114:0/2403125040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b240f69b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 -- 172.21.15.114:0/2403125040 >> 172.21.15.114:0/2403125040 conn(0x7f4b240f1cd0 msgr2=0x7f4b240f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 -- 172.21.15.114:0/2403125040 shutdown_connections 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 -- 172.21.15.114:0/2403125040 wait complete. 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.835+0000 7f4b2a577700 1 Processor -- start 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b2a577700 1 -- start start 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b2a577700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b241067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b2a577700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b24106cb0 con 0x7f4b240f6590 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b241067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b241067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54902/0 (socket says 172.21.15.114:54902) 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b23fff700 1 -- 172.21.15.114:0/3892300312 learned_addr learned my addr 172.21.15.114:0/3892300312 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b23fff700 1 -- 172.21.15.114:0/3892300312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b14009d70 con 0x7f4b240f6590 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.836+0000 7f4b23fff700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b241067c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f4b14004170 tx=0x7f4b14009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.837+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b1400e040 con 0x7f4b240f6590 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.837+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b24106eb0 con 0x7f4b240f6590 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.837+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4b14008ef0 con 0x7f4b240f6590 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.837+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b14014c90 con 0x7f4b240f6590 2024-02-20T13:18:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.837+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b24107340 con 0x7f4b240f6590 2024-02-20T13:18:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.837+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4b140283f0 con 0x7f4b240f6590 2024-02-20T13:18:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.838+0000 7f4b20ff9700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b0c031090 0x7f4b0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.838+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b24045bb0 con 0x7f4b240f6590 2024-02-20T13:18:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.838+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4b14042210 con 0x7f4b240f6590 2024-02-20T13:18:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.838+0000 7f4b237fe700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b0c031090 0x7f4b0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:43.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.838+0000 7f4b237fe700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b0c031090 0x7f4b0c033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4b180096a0 tx=0x7f4b18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:43.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:43.843+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4b14026300 con 0x7f4b240f6590 2024-02-20T13:18:44.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.047+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b241081f0 con 0x7f4b240f6590 2024-02-20T13:18:44.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.048+0000 7f4b20ff9700 1 -- 172.21.15.114:0/3892300312 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b14012b20 con 0x7f4b240f6590 2024-02-20T13:18:44.050 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:44.050 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b0c031090 msgr2=0x7f4b0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b0c031090 0x7f4b0c033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4b180096a0 tx=0x7f4b18006b40).stop 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 msgr2=0x7f4b241067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b241067c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f4b14004170 tx=0x7f4b14009ad0).stop 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 shutdown_connections 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b0c031090 0x7f4b0c033550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 --2- 172.21.15.114:0/3892300312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b240f6590 0x7f4b241067c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 >> 172.21.15.114:0/3892300312 conn(0x7f4b240f1cd0 msgr2=0x7f4b240f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 shutdown_connections 2024-02-20T13:18:44.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:44.050+0000 7f4b2a577700 1 -- 172.21.15.114:0/3892300312 wait complete. 2024-02-20T13:18:44.053 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:44.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:44.049833+0000 mon.smithi038 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.114:0/3892300312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:45.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:44.145767+0000 mgr.smithi038.powepi (mgr.14156) 40 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:45.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:45.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:47.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 -- 172.21.15.114:0/3174889122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 msgr2=0x7f42600f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 --2- 172.21.15.114:0/3174889122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42600f58a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4250009960 tx=0x7f4250004d10).stop 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 -- 172.21.15.114:0/3174889122 shutdown_connections 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 --2- 172.21.15.114:0/3174889122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42600f58a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 -- 172.21.15.114:0/3174889122 >> 172.21.15.114:0/3174889122 conn(0x7f42600f0be0 msgr2=0x7f42600f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 -- 172.21.15.114:0/3174889122 shutdown_connections 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.386+0000 7f4267cdc700 1 -- 172.21.15.114:0/3174889122 wait complete. 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.387+0000 7f4267cdc700 1 Processor -- start 2024-02-20T13:18:47.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.387+0000 7f4267cdc700 1 -- start start 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.387+0000 7f4267cdc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42601094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.387+0000 7f4267cdc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260060320 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4265a78700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42601094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4265a78700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42601094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35182/0 (socket says 172.21.15.114:35182) 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4265a78700 1 -- 172.21.15.114:0/1397276595 learned_addr learned my addr 172.21.15.114:0/1397276595 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4265a78700 1 -- 172.21.15.114:0/1397276595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250009640 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4265a78700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42601094f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4250009610 tx=0x7f4250004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4250018070 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260060520 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f425000b7e0 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425001d420 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.388+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260109ba0 con 0x7f42600f5480 2024-02-20T13:18:47.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.389+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4250024060 con 0x7f42600f5480 2024-02-20T13:18:47.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.389+0000 7f425e7fc700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4248031090 0x7f4248033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:47.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.389+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260045bb0 con 0x7f42600f5480 2024-02-20T13:18:47.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.389+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4250042630 con 0x7f42600f5480 2024-02-20T13:18:47.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.389+0000 7f4265277700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4248031090 0x7f4248033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:47.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.390+0000 7f4265277700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4248031090 0x7f4248033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f42540096a0 tx=0x7f4254006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:47.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.394+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f42500116a0 con 0x7f42600f5480 2024-02-20T13:18:47.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:46.146079+0000 mgr.smithi038.powepi (mgr.14156) 41 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:47.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.597+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42600fb060 con 0x7f42600f5480 2024-02-20T13:18:47.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.598+0000 7f425e7fc700 1 -- 172.21.15.114:0/1397276595 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42500118b0 con 0x7f42600f5480 2024-02-20T13:18:47.600 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:47.600 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:47.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4248031090 msgr2=0x7f4248033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:47.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4248031090 0x7f4248033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f42540096a0 tx=0x7f4254006b40).stop 2024-02-20T13:18:47.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 msgr2=0x7f42601094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42601094f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4250009610 tx=0x7f4250004d10).stop 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 shutdown_connections 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4248031090 0x7f4248033550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 --2- 172.21.15.114:0/1397276595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42600f5480 0x7f42601094f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 >> 172.21.15.114:0/1397276595 conn(0x7f42600f0be0 msgr2=0x7f42600f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 shutdown_connections 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:47.600+0000 7f4267cdc700 1 -- 172.21.15.114:0/1397276595 wait complete. 2024-02-20T13:18:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:48.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:47.599883+0000 mon.smithi038 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.114:0/1397276595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:49.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:49.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:49.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:48.146368+0000 mgr.smithi038.powepi (mgr.14156) 42 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:51.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- 172.21.15.114:0/1141278284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 msgr2=0x7f8e240f4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 --2- 172.21.15.114:0/1141278284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e240f4e90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f8e20009fb0 tx=0x7f8e20009b30).stop 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- 172.21.15.114:0/1141278284 shutdown_connections 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 --2- 172.21.15.114:0/1141278284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e240f4e90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- 172.21.15.114:0/1141278284 >> 172.21.15.114:0/1141278284 conn(0x7f8e240f01d0 msgr2=0x7f8e240f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- 172.21.15.114:0/1141278284 shutdown_connections 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- 172.21.15.114:0/1141278284 wait complete. 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 Processor -- start 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- start start 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e2410b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.104+0000 7f8e2aa63700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e24107120 con 0x7f8e240f4a70 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e29a61700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e2410b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e29a61700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e2410b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35188/0 (socket says 172.21.15.114:35188) 2024-02-20T13:18:51.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e29a61700 1 -- 172.21.15.114:0/4214184772 learned_addr learned my addr 172.21.15.114:0/4214184772 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e29a61700 1 -- 172.21.15.114:0/4214184772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e20005040 con 0x7f8e240f4a70 2024-02-20T13:18:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e29a61700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e2410b730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8e200089f0 tx=0x7f8e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e20009320 con 0x7f8e240f4a70 2024-02-20T13:18:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e240f17f0 con 0x7f8e240f4a70 2024-02-20T13:18:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e20013ec0 con 0x7f8e240f4a70 2024-02-20T13:18:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.105+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e20014d30 con 0x7f8e240f4a70 2024-02-20T13:18:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.106+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2410bc20 con 0x7f8e240f4a70 2024-02-20T13:18:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.106+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8e200095f0 con 0x7f8e240f4a70 2024-02-20T13:18:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.106+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e24045bb0 con 0x7f8e240f4a70 2024-02-20T13:18:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.106+0000 7f8e1a7fc700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8e10031090 0x7f8e10033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.107+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e200424a0 con 0x7f8e240f4a70 2024-02-20T13:18:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.107+0000 7f8e29260700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8e10031090 0x7f8e10033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.107+0000 7f8e29260700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8e10031090 0x7f8e10033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8e140096a0 tx=0x7f8e14006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.111+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e2001d7f0 con 0x7f8e240f4a70 2024-02-20T13:18:51.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.316+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e24027ae0 con 0x7f8e240f4a70 2024-02-20T13:18:51.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.317+0000 7f8e1a7fc700 1 -- 172.21.15.114:0/4214184772 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e2001d7f0 con 0x7f8e240f4a70 2024-02-20T13:18:51.319 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:51.319 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8e10031090 msgr2=0x7f8e10033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8e10031090 0x7f8e10033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8e140096a0 tx=0x7f8e14006b40).stop 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 msgr2=0x7f8e2410b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e2410b730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8e200089f0 tx=0x7f8e2000b040).stop 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 shutdown_connections 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8e10031090 0x7f8e10033550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 --2- 172.21.15.114:0/4214184772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e240f4a70 0x7f8e2410b730 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 >> 172.21.15.114:0/4214184772 conn(0x7f8e240f01d0 msgr2=0x7f8e240f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 shutdown_connections 2024-02-20T13:18:51.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:51.319+0000 7f8e2aa63700 1 -- 172.21.15.114:0/4214184772 wait complete. 2024-02-20T13:18:51.322 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:51.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:50.146703+0000 mgr.smithi038.powepi (mgr.14156) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:52.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:51.318781+0000 mon.smithi038 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.114:0/4214184772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:52.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:52.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:53.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:52.146972+0000 mgr.smithi038.powepi (mgr.14156) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 -- 172.21.15.114:0/1901297018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 msgr2=0x7f9b780f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 --2- 172.21.15.114:0/1901297018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b780f6800 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9b74009fb0 tx=0x7f9b74009b30).stop 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 -- 172.21.15.114:0/1901297018 shutdown_connections 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 --2- 172.21.15.114:0/1901297018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b780f6800 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 -- 172.21.15.114:0/1901297018 >> 172.21.15.114:0/1901297018 conn(0x7f9b780f1b40 msgr2=0x7f9b780f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 -- 172.21.15.114:0/1901297018 shutdown_connections 2024-02-20T13:18:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.744+0000 7f9b7eb33700 1 -- 172.21.15.114:0/1901297018 wait complete. 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7eb33700 1 Processor -- start 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7eb33700 1 -- start start 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7eb33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b78106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7eb33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78106b10 con 0x7f9b780f63e0 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7db31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b78106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7db31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b78106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35206/0 (socket says 172.21.15.114:35206) 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7db31700 1 -- 172.21.15.114:0/4121913969 learned_addr learned my addr 172.21.15.114:0/4121913969 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7db31700 1 -- 172.21.15.114:0/4121913969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b74005040 con 0x7f9b780f63e0 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.745+0000 7f9b7db31700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b78106620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9b740089f0 tx=0x7f9b7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.746+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b74009320 con 0x7f9b780f63e0 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.746+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b78106d10 con 0x7f9b780f63e0 2024-02-20T13:18:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.746+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9b74013ec0 con 0x7f9b780f63e0 2024-02-20T13:18:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.746+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b74014d30 con 0x7f9b780f63e0 2024-02-20T13:18:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.746+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781071a0 con 0x7f9b780f63e0 2024-02-20T13:18:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.746+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9b740095f0 con 0x7f9b780f63e0 2024-02-20T13:18:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.747+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b78045bb0 con 0x7f9b780f63e0 2024-02-20T13:18:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.747+0000 7f9b6e7fc700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b64031090 0x7f9b64033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.747+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9b740424a0 con 0x7f9b780f63e0 2024-02-20T13:18:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.747+0000 7f9b7d330700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b64031090 0x7f9b64033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.747+0000 7f9b7d330700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b64031090 0x7f9b64033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9b680096a0 tx=0x7f9b68006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.752+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9b7401d980 con 0x7f9b780f63e0 2024-02-20T13:18:54.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.959+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b78108050 con 0x7f9b780f63e0 2024-02-20T13:18:54.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.959+0000 7f9b6e7fc700 1 -- 172.21.15.114:0/4121913969 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b7401d380 con 0x7f9b780f63e0 2024-02-20T13:18:54.961 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:54.961 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b64031090 msgr2=0x7f9b64033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b64031090 0x7f9b64033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9b680096a0 tx=0x7f9b68006b40).stop 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 msgr2=0x7f9b78106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b78106620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9b740089f0 tx=0x7f9b7400b040).stop 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 shutdown_connections 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b64031090 0x7f9b64033550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 --2- 172.21.15.114:0/4121913969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b780f63e0 0x7f9b78106620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 >> 172.21.15.114:0/4121913969 conn(0x7f9b780f1b40 msgr2=0x7f9b780f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 shutdown_connections 2024-02-20T13:18:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:54.962+0000 7f9b7eb33700 1 -- 172.21.15.114:0/4121913969 wait complete. 2024-02-20T13:18:54.964 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:55.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:54.147252+0000 mgr.smithi038.powepi (mgr.14156) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:55.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:54.961047+0000 mon.smithi038 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.114:0/4121913969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:18:56.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:18:56.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:18:57.447 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:56.147567+0000 mgr.smithi038.powepi (mgr.14156) 46 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.554+0000 7f59cb252700 1 -- 172.21.15.114:0/4044915225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 msgr2=0x7f59c40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.554+0000 7f59cb252700 1 --2- 172.21.15.114:0/4044915225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c40f69b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f59b8004760 tx=0x7f59b8009ad0).stop 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.554+0000 7f59cb252700 1 -- 172.21.15.114:0/4044915225 shutdown_connections 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.554+0000 7f59cb252700 1 --2- 172.21.15.114:0/4044915225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c40f69b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.554+0000 7f59cb252700 1 -- 172.21.15.114:0/4044915225 >> 172.21.15.114:0/4044915225 conn(0x7f59c40f1cd0 msgr2=0x7f59c40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.555+0000 7f59cb252700 1 -- 172.21.15.114:0/4044915225 shutdown_connections 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.555+0000 7f59cb252700 1 -- 172.21.15.114:0/4044915225 wait complete. 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.555+0000 7f59cb252700 1 Processor -- start 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.555+0000 7f59cb252700 1 -- start start 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.555+0000 7f59cb252700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.555+0000 7f59cb252700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4069f00 con 0x7f59c40f6590 2024-02-20T13:18:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c8fee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c8fee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:45784/0 (socket says 172.21.15.114:45784) 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c8fee700 1 -- 172.21.15.114:0/3850175648 learned_addr learned my addr 172.21.15.114:0/3850175648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c8fee700 1 -- 172.21.15.114:0/3850175648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8009d70 con 0x7f59c40f6590 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c8fee700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c406ba80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f59b8004170 tx=0x7f59b8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b800e040 con 0x7f59c40f6590 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c406a0a0 con 0x7f59c40f6590 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f59b8008ef0 con 0x7f59c40f6590 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b8014c90 con 0x7f59c40f6590 2024-02-20T13:18:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.556+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c406a530 con 0x7f59c40f6590 2024-02-20T13:18:58.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.557+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f59b80283f0 con 0x7f59c40f6590 2024-02-20T13:18:58.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.557+0000 7f59c17fa700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f59b4031090 0x7f59b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:18:58.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.557+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f59b80421f0 con 0x7f59c40f6590 2024-02-20T13:18:58.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.557+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c40591e0 con 0x7f59c40f6590 2024-02-20T13:18:58.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.557+0000 7f59c3fff700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f59b4031090 0x7f59b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:18:58.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.558+0000 7f59c3fff700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f59b4031090 0x7f59b4033550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f59b00096a0 tx=0x7f59b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:18:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.562+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f59b80268f0 con 0x7f59c40f6590 2024-02-20T13:18:58.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.769+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59c406ab90 con 0x7f59c40f6590 2024-02-20T13:18:58.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.769+0000 7f59c17fa700 1 -- 172.21.15.114:0/3850175648 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59b8026300 con 0x7f59c40f6590 2024-02-20T13:18:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:18:58.770 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f59b4031090 msgr2=0x7f59b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f59b4031090 0x7f59b4033550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f59b00096a0 tx=0x7f59b0006b40).stop 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 msgr2=0x7f59c406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c406ba80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f59b8004170 tx=0x7f59b8009ad0).stop 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 shutdown_connections 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f59b4031090 0x7f59b4033550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 --2- 172.21.15.114:0/3850175648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59c40f6590 0x7f59c406ba80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:18:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 >> 172.21.15.114:0/3850175648 conn(0x7f59c40f1cd0 msgr2=0x7f59c40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:18:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 shutdown_connections 2024-02-20T13:18:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:18:58.771+0000 7f59cb252700 1 -- 172.21.15.114:0/3850175648 wait complete. 2024-02-20T13:18:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:18:59.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:18:58.147839+0000 mgr.smithi038.powepi (mgr.14156) 47 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:18:59.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:18:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:18:58.770041+0000 mon.smithi038 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.114:0/3850175648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:00.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:00.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:01.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:00.148099+0000 mgr.smithi038.powepi (mgr.14156) 48 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 -- 172.21.15.114:0/3385396628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 msgr2=0x7f1fc80f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 --2- 172.21.15.114:0/3385396628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f9160 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1fb8009fb0 tx=0x7f1fb8009b30).stop 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 -- 172.21.15.114:0/3385396628 shutdown_connections 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 --2- 172.21.15.114:0/3385396628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f9160 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 -- 172.21.15.114:0/3385396628 >> 172.21.15.114:0/3385396628 conn(0x7f1fc80f0be0 msgr2=0x7f1fc80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 -- 172.21.15.114:0/3385396628 shutdown_connections 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.348+0000 7f1fcdb48700 1 -- 172.21.15.114:0/3385396628 wait complete. 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fcdb48700 1 Processor -- start 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fcdb48700 1 -- start start 2024-02-20T13:19:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fcdb48700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fcdb48700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80f5060 con 0x7f1fc8060360 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fc77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fc77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:45808/0 (socket says 172.21.15.114:45808) 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.349+0000 7f1fc77fe700 1 -- 172.21.15.114:0/1164666916 learned_addr learned my addr 172.21.15.114:0/1164666916 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1fc77fe700 1 -- 172.21.15.114:0/1164666916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb8005040 con 0x7f1fc8060360 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1fc77fe700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f6be0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1fb8009f80 tx=0x7f1fb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb8009320 con 0x7f1fc8060360 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc80f5200 con 0x7f1fc8060360 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1fb8013680 con 0x7f1fc8060360 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb801eda0 con 0x7f1fc8060360 2024-02-20T13:19:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.350+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc80f5690 con 0x7f1fc8060360 2024-02-20T13:19:02.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.351+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1fb801e420 con 0x7f1fc8060360 2024-02-20T13:19:02.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.351+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc80591e0 con 0x7f1fc8060360 2024-02-20T13:19:02.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.351+0000 7f1faffff700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fb00310e0 0x7f1fb00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:02.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.351+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1fb8042350 con 0x7f1fc8060360 2024-02-20T13:19:02.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.351+0000 7f1fc6ffd700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fb00310e0 0x7f1fb00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:02.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.352+0000 7f1fc6ffd700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fb00310e0 0x7f1fb00335a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1fbc0096a0 tx=0x7f1fbc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:02.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.356+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1fb801e6b0 con 0x7f1fc8060360 2024-02-20T13:19:02.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.564+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc8045bb0 con 0x7f1fc8060360 2024-02-20T13:19:02.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.565+0000 7f1faffff700 1 -- 172.21.15.114:0/1164666916 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fb802c2d0 con 0x7f1fc8060360 2024-02-20T13:19:02.568 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:02.568 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fb00310e0 msgr2=0x7f1fb00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fb00310e0 0x7f1fb00335a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1fbc0096a0 tx=0x7f1fbc006b40).stop 2024-02-20T13:19:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 msgr2=0x7f1fc80f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f6be0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1fb8009f80 tx=0x7f1fb800b040).stop 2024-02-20T13:19:02.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 shutdown_connections 2024-02-20T13:19:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fb00310e0 0x7f1fb00335a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 --2- 172.21.15.114:0/1164666916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc8060360 0x7f1fc80f6be0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 >> 172.21.15.114:0/1164666916 conn(0x7f1fc80f0be0 msgr2=0x7f1fc80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 shutdown_connections 2024-02-20T13:19:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:02.567+0000 7f1fcdb48700 1 -- 172.21.15.114:0/1164666916 wait complete. 2024-02-20T13:19:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:02.148431+0000 mgr.smithi038.powepi (mgr.14156) 49 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:02.565777+0000 mon.smithi038 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.114:0/1164666916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:04.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:04.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:05.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:04.148727+0000 mgr.smithi038.powepi (mgr.14156) 50 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/1214093634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 msgr2=0x7f4b380f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 --2- 172.21.15.114:0/1214093634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b380f56f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4b2c009fb0 tx=0x7f4b2c009b30).stop 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/1214093634 shutdown_connections 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 --2- 172.21.15.114:0/1214093634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b380f56f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/1214093634 >> 172.21.15.114:0/1214093634 conn(0x7f4b380f0a30 msgr2=0x7f4b380f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/1214093634 shutdown_connections 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/1214093634 wait complete. 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 Processor -- start 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.043+0000 7f4b3d9a1700 1 -- start start 2024-02-20T13:19:06.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3d9a1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b38109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3d9a1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b38060320 con 0x7f4b380f52d0 2024-02-20T13:19:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3c99f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b38109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3c99f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b38109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:40292/0 (socket says 172.21.15.114:40292) 2024-02-20T13:19:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3c99f700 1 -- 172.21.15.114:0/3683969841 learned_addr learned my addr 172.21.15.114:0/3683969841 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3c99f700 1 -- 172.21.15.114:0/3683969841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c005040 con 0x7f4b380f52d0 2024-02-20T13:19:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.044+0000 7f4b3c99f700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b38109310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4b2c000c00 tx=0x7f4b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:06.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.045+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b2c0095f0 con 0x7f4b380f52d0 2024-02-20T13:19:06.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.045+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b38060520 con 0x7f4b380f52d0 2024-02-20T13:19:06.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.045+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4b2c007e50 con 0x7f4b380f52d0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.045+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b2c014e70 con 0x7f4b380f52d0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.045+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b381099c0 con 0x7f4b380f52d0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.045+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4b2c025070 con 0x7f4b380f52d0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.046+0000 7f4b357fa700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b28031090 0x7f4b28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.046+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b38045bb0 con 0x7f4b380f52d0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.046+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4b2c041d80 con 0x7f4b380f52d0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.046+0000 7f4b37fff700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b28031090 0x7f4b28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:06.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.046+0000 7f4b37fff700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b28031090 0x7f4b28033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4b240096a0 tx=0x7f4b24006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:06.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.051+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4b2c0079c0 con 0x7f4b380f52d0 2024-02-20T13:19:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.258+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b3810a1b0 con 0x7f4b380f52d0 2024-02-20T13:19:06.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.258+0000 7f4b357fa700 1 -- 172.21.15.114:0/3683969841 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b2c00f300 con 0x7f4b380f52d0 2024-02-20T13:19:06.259 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:06.259 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b28031090 msgr2=0x7f4b28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b28031090 0x7f4b28033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4b240096a0 tx=0x7f4b24006b40).stop 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 msgr2=0x7f4b38109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b38109310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4b2c000c00 tx=0x7f4b2c00b040).stop 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 shutdown_connections 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4b28031090 0x7f4b28033550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 --2- 172.21.15.114:0/3683969841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b380f52d0 0x7f4b38109310 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 >> 172.21.15.114:0/3683969841 conn(0x7f4b380f0a30 msgr2=0x7f4b380f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:06.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 shutdown_connections 2024-02-20T13:19:06.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:06.260+0000 7f4b3d9a1700 1 -- 172.21.15.114:0/3683969841 wait complete. 2024-02-20T13:19:06.262 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:07.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:06.148997+0000 mgr.smithi038.powepi (mgr.14156) 51 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:07.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:06.259038+0000 mon.smithi038 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.114:0/3683969841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:07.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:07.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:09.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:08.149350+0000 mgr.smithi038.powepi (mgr.14156) 52 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:09.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 -- 172.21.15.114:0/1940660296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 msgr2=0x7fa0100f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:09.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 --2- 172.21.15.114:0/1940660296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0100f58a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa004009fb0 tx=0x7fa004009b30).stop 2024-02-20T13:19:09.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 -- 172.21.15.114:0/1940660296 shutdown_connections 2024-02-20T13:19:09.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 --2- 172.21.15.114:0/1940660296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0100f58a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:09.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 -- 172.21.15.114:0/1940660296 >> 172.21.15.114:0/1940660296 conn(0x7fa0100f0be0 msgr2=0x7fa0100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 -- 172.21.15.114:0/1940660296 shutdown_connections 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.756+0000 7fa015105700 1 -- 172.21.15.114:0/1940660296 wait complete. 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa015105700 1 Processor -- start 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa015105700 1 -- start start 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa015105700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0101094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa015105700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa010060320 con 0x7fa0100f5480 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa00ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0101094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa00ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0101094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:40322/0 (socket says 172.21.15.114:40322) 2024-02-20T13:19:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa00ed9d700 1 -- 172.21.15.114:0/1399545261 learned_addr learned my addr 172.21.15.114:0/1399545261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.757+0000 7fa00ed9d700 1 -- 172.21.15.114:0/1399545261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004005040 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.758+0000 7fa00ed9d700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0101094f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa0040089f0 tx=0x7fa00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.758+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa004009320 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.758+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa004013ec0 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.758+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa010060520 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.758+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa004014d30 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.758+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa010109ba0 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.759+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa0040095f0 con 0x7fa0100f5480 2024-02-20T13:19:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.759+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa010045bb0 con 0x7fa0100f5480 2024-02-20T13:19:09.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.759+0000 7f9ff77fe700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9ff8031090 0x7f9ff8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:09.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.759+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa0040423e0 con 0x7fa0100f5480 2024-02-20T13:19:09.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.759+0000 7fa00e59c700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9ff8031090 0x7f9ff8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:09.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.759+0000 7fa00e59c700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9ff8031090 0x7f9ff8033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa0000096a0 tx=0x7fa000006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:09.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.764+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa00401d7f0 con 0x7fa0100f5480 2024-02-20T13:19:09.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.969+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0100faf70 con 0x7fa0100f5480 2024-02-20T13:19:09.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.970+0000 7f9ff77fe700 1 -- 172.21.15.114:0/1399545261 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa00401d7f0 con 0x7fa0100f5480 2024-02-20T13:19:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:09.972 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9ff8031090 msgr2=0x7f9ff8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9ff8031090 0x7f9ff8033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa0000096a0 tx=0x7fa000006b40).stop 2024-02-20T13:19:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 msgr2=0x7fa0101094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0101094f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa0040089f0 tx=0x7fa00400b040).stop 2024-02-20T13:19:09.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 shutdown_connections 2024-02-20T13:19:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9ff8031090 0x7f9ff8033550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 --2- 172.21.15.114:0/1399545261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0100f5480 0x7fa0101094f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 >> 172.21.15.114:0/1399545261 conn(0x7fa0100f0be0 msgr2=0x7fa0100f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 shutdown_connections 2024-02-20T13:19:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:09.972+0000 7fa015105700 1 -- 172.21.15.114:0/1399545261 wait complete. 2024-02-20T13:19:09.975 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:10.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:09.970798+0000 mon.smithi038 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.114:0/1399545261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:11.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:10.149656+0000 mgr.smithi038.powepi (mgr.14156) 53 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:11.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:11.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:13.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- 172.21.15.114:0/921754928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 msgr2=0x7f08e80f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:13.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 --2- 172.21.15.114:0/921754928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e80f8fb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f08e4009fb0 tx=0x7f08e4009b30).stop 2024-02-20T13:19:13.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- 172.21.15.114:0/921754928 shutdown_connections 2024-02-20T13:19:13.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 --2- 172.21.15.114:0/921754928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e80f8fb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:13.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- 172.21.15.114:0/921754928 >> 172.21.15.114:0/921754928 conn(0x7f08e80f0a10 msgr2=0x7f08e80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:13.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- 172.21.15.114:0/921754928 shutdown_connections 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- 172.21.15.114:0/921754928 wait complete. 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 Processor -- start 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- start start 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e81065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.407+0000 7f08eea32700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8106aa0 con 0x7f08e8060360 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eda30700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e81065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eda30700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e81065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:40354/0 (socket says 172.21.15.114:40354) 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eda30700 1 -- 172.21.15.114:0/1145533205 learned_addr learned my addr 172.21.15.114:0/1145533205 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eda30700 1 -- 172.21.15.114:0/1145533205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4005040 con 0x7f08e8060360 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eda30700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e81065b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f08e4009f80 tx=0x7f08e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08e4009320 con 0x7f08e8060360 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e8106ca0 con 0x7f08e8060360 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f08e4013680 con 0x7f08e8060360 2024-02-20T13:19:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08e401eda0 con 0x7f08e8060360 2024-02-20T13:19:13.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.408+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e8107130 con 0x7f08e8060360 2024-02-20T13:19:13.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.409+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f08e401e420 con 0x7f08e8060360 2024-02-20T13:19:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.409+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8045bb0 con 0x7f08e8060360 2024-02-20T13:19:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.409+0000 7f08de7fc700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f08d40310e0 0x7f08d40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.409+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f08e4042410 con 0x7f08e8060360 2024-02-20T13:19:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.409+0000 7f08ed22f700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f08d40310e0 0x7f08d40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.410+0000 7f08ed22f700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f08d40310e0 0x7f08d40335a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f08d80096a0 tx=0x7f08d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:13.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.414+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f08e401ccf0 con 0x7f08e8060360 2024-02-20T13:19:13.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:12.149988+0000 mgr.smithi038.powepi (mgr.14156) 54 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.619+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08e8107fe0 con 0x7f08e8060360 2024-02-20T13:19:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.620+0000 7f08de7fc700 1 -- 172.21.15.114:0/1145533205 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f08e401cb10 con 0x7f08e8060360 2024-02-20T13:19:13.621 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:13.621 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f08d40310e0 msgr2=0x7f08d40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f08d40310e0 0x7f08d40335a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f08d80096a0 tx=0x7f08d8006b40).stop 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 msgr2=0x7f08e81065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e81065b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f08e4009f80 tx=0x7f08e400b040).stop 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 shutdown_connections 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f08d40310e0 0x7f08d40335a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:13.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 --2- 172.21.15.114:0/1145533205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e8060360 0x7f08e81065b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:13.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 >> 172.21.15.114:0/1145533205 conn(0x7f08e80f0a10 msgr2=0x7f08e80f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:13.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 shutdown_connections 2024-02-20T13:19:13.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:13.621+0000 7f08eea32700 1 -- 172.21.15.114:0/1145533205 wait complete. 2024-02-20T13:19:13.623 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:14.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:13.620472+0000 mon.smithi038 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.114:0/1145533205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:15.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:15.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:15.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:14.150262+0000 mgr.smithi038.powepi (mgr.14156) 55 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:17.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 -- 172.21.15.114:0/1357565006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 msgr2=0x7f15240f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:17.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 --2- 172.21.15.114:0/1357565006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15240f69b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f1514004760 tx=0x7f1514009ad0).stop 2024-02-20T13:19:17.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 -- 172.21.15.114:0/1357565006 shutdown_connections 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 --2- 172.21.15.114:0/1357565006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15240f69b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 -- 172.21.15.114:0/1357565006 >> 172.21.15.114:0/1357565006 conn(0x7f15240f1cd0 msgr2=0x7f15240f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 -- 172.21.15.114:0/1357565006 shutdown_connections 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.183+0000 7f152bed2700 1 -- 172.21.15.114:0/1357565006 wait complete. 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f152bed2700 1 Processor -- start 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f152bed2700 1 -- start start 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f152bed2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15241067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f152bed2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524106cb0 con 0x7f15240f6590 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f1529c6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15241067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f1529c6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15241067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54072/0 (socket says 172.21.15.114:54072) 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f1529c6e700 1 -- 172.21.15.114:0/388388971 learned_addr learned my addr 172.21.15.114:0/388388971 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f1529c6e700 1 -- 172.21.15.114:0/388388971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514009d70 con 0x7f15240f6590 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.184+0000 7f1529c6e700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15241067c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1514013f10 tx=0x7f1514009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:17.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.185+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f151400e040 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.185+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1524106eb0 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.185+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1514007df0 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.185+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1524107340 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.185+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1514014dd0 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.186+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1514014520 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.186+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1524045bb0 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.186+0000 7f151a7fc700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1510031040 0x7f1510033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.186+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f15140424f0 con 0x7f15240f6590 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.186+0000 7f152946d700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1510031040 0x7f1510033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.186+0000 7f152946d700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1510031040 0x7f1510033500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f15200096a0 tx=0x7f1520006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:17.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.191+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1514007960 con 0x7f15240f6590 2024-02-20T13:19:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.396+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15241081f0 con 0x7f15240f6590 2024-02-20T13:19:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.397+0000 7f151a7fc700 1 -- 172.21.15.114:0/388388971 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1514027e20 con 0x7f15240f6590 2024-02-20T13:19:17.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:17.398 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:17.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1510031040 msgr2=0x7f1510033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1510031040 0x7f1510033500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f15200096a0 tx=0x7f1520006b40).stop 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 msgr2=0x7f15241067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15241067c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1514013f10 tx=0x7f1514009ad0).stop 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 shutdown_connections 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1510031040 0x7f1510033500 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 --2- 172.21.15.114:0/388388971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15240f6590 0x7f15241067c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 >> 172.21.15.114:0/388388971 conn(0x7f15240f1cd0 msgr2=0x7f15240f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 shutdown_connections 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:17.399+0000 7f152bed2700 1 -- 172.21.15.114:0/388388971 wait complete. 2024-02-20T13:19:17.400 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:17.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:16.150601+0000 mgr.smithi038.powepi (mgr.14156) 56 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:18.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:17.397803+0000 mon.smithi038 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.114:0/388388971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:18.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:18.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:19.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:18.150942+0000 mgr.smithi038.powepi (mgr.14156) 57 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:20.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- 172.21.15.114:0/2196681499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 msgr2=0x7f29d40f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:20.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 --2- 172.21.15.114:0/2196681499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d40f6800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f29c4009fb0 tx=0x7f29c4009b30).stop 2024-02-20T13:19:20.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- 172.21.15.114:0/2196681499 shutdown_connections 2024-02-20T13:19:20.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 --2- 172.21.15.114:0/2196681499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d40f6800 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:20.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- 172.21.15.114:0/2196681499 >> 172.21.15.114:0/2196681499 conn(0x7f29d40f1b40 msgr2=0x7f29d40f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- 172.21.15.114:0/2196681499 shutdown_connections 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- 172.21.15.114:0/2196681499 wait complete. 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 Processor -- start 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- start start 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d4106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.792+0000 7f29d359e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d4106b10 con 0x7f29d40f63e0 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d4106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d4106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54092/0 (socket says 172.21.15.114:54092) 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d259c700 1 -- 172.21.15.114:0/2744766782 learned_addr learned my addr 172.21.15.114:0/2744766782 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d259c700 1 -- 172.21.15.114:0/2744766782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c4005040 con 0x7f29d40f63e0 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d259c700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d4106620 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f29c40089f0 tx=0x7f29c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c4009320 con 0x7f29d40f63e0 2024-02-20T13:19:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f29c4013ec0 con 0x7f29d40f63e0 2024-02-20T13:19:20.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d4106d10 con 0x7f29d40f63e0 2024-02-20T13:19:20.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c4014d30 con 0x7f29d40f63e0 2024-02-20T13:19:20.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.793+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d41071a0 con 0x7f29d40f63e0 2024-02-20T13:19:20.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.794+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f29c40095f0 con 0x7f29d40f63e0 2024-02-20T13:19:20.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.794+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d4045bb0 con 0x7f29d40f63e0 2024-02-20T13:19:20.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.794+0000 7f29c2ffd700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f29bc0310e0 0x7f29bc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:20.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.794+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f29c40423a0 con 0x7f29d40f63e0 2024-02-20T13:19:20.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.794+0000 7f29d1d9b700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f29bc0310e0 0x7f29bc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:20.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.795+0000 7f29d1d9b700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f29bc0310e0 0x7f29bc0335a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f29c80096a0 tx=0x7f29c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:20.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:20.799+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f29c401d7f0 con 0x7f29d40f63e0 2024-02-20T13:19:21.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.006+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29d4108050 con 0x7f29d40f63e0 2024-02-20T13:19:21.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.007+0000 7f29c2ffd700 1 -- 172.21.15.114:0/2744766782 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29c401d7f0 con 0x7f29d40f63e0 2024-02-20T13:19:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:21.009 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f29bc0310e0 msgr2=0x7f29bc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f29bc0310e0 0x7f29bc0335a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f29c80096a0 tx=0x7f29c8006b40).stop 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 msgr2=0x7f29d4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d4106620 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f29c40089f0 tx=0x7f29c400b040).stop 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 shutdown_connections 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f29bc0310e0 0x7f29bc0335a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 --2- 172.21.15.114:0/2744766782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29d40f63e0 0x7f29d4106620 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 >> 172.21.15.114:0/2744766782 conn(0x7f29d40f1b40 msgr2=0x7f29d40f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 shutdown_connections 2024-02-20T13:19:21.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:21.009+0000 7f29d359e700 1 -- 172.21.15.114:0/2744766782 wait complete. 2024-02-20T13:19:21.012 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:21.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:20.151244+0000 mgr.smithi038.powepi (mgr.14156) 58 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:21.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:21.008822+0000 mon.smithi038 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.114:0/2744766782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:22.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:22.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:23.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:22.151615+0000 mgr.smithi038.powepi (mgr.14156) 59 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:24.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 -- 172.21.15.114:0/4202530502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 msgr2=0x7f7bc40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 --2- 172.21.15.114:0/4202530502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40f58a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7bb4009fb0 tx=0x7f7bb4009b30).stop 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 -- 172.21.15.114:0/4202530502 shutdown_connections 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 --2- 172.21.15.114:0/4202530502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40f58a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 -- 172.21.15.114:0/4202530502 >> 172.21.15.114:0/4202530502 conn(0x7f7bc40f0be0 msgr2=0x7f7bc40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 -- 172.21.15.114:0/4202530502 shutdown_connections 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.598+0000 7f7bcc549700 1 -- 172.21.15.114:0/4202530502 wait complete. 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bcc549700 1 Processor -- start 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bcc549700 1 -- start start 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bcc549700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bcc549700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40fb8e0 con 0x7f7bc40f5480 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bca2e5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bca2e5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54104/0 (socket says 172.21.15.114:54104) 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bca2e5700 1 -- 172.21.15.114:0/1912532783 learned_addr learned my addr 172.21.15.114:0/1912532783 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bca2e5700 1 -- 172.21.15.114:0/1912532783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb4005040 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.599+0000 7f7bca2e5700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40fd460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7bb4000c00 tx=0x7f7bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb40095f0 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc40fba80 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7bb4007e50 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb4014e70 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc40fbf10 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7bb4025070 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.600+0000 7f7bbaffd700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7bb0031090 0x7f7bb0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.601+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7bb4041d60 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.601+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc40591e0 con 0x7f7bc40f5480 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.601+0000 7f7bc9ae4700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7bb0031090 0x7f7bb0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.601+0000 7f7bc9ae4700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7bb0031090 0x7f7bb0033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7bc00096a0 tx=0x7f7bc0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.606+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7bb40146b0 con 0x7f7bc40f5480 2024-02-20T13:19:24.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.811+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bc4045bb0 con 0x7f7bc40f5480 2024-02-20T13:19:24.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.812+0000 7f7bbaffd700 1 -- 172.21.15.114:0/1912532783 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bb402d090 con 0x7f7bc40f5480 2024-02-20T13:19:24.814 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:24.815 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7bb0031090 msgr2=0x7f7bb0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7bb0031090 0x7f7bb0033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7bc00096a0 tx=0x7f7bc0006b40).stop 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 msgr2=0x7f7bc40fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40fd460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7bb4000c00 tx=0x7f7bb400b040).stop 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 shutdown_connections 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7bb0031090 0x7f7bb0033550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 --2- 172.21.15.114:0/1912532783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7bc40f5480 0x7f7bc40fd460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 >> 172.21.15.114:0/1912532783 conn(0x7f7bc40f0be0 msgr2=0x7f7bc405fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 shutdown_connections 2024-02-20T13:19:24.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:24.814+0000 7f7bcc549700 1 -- 172.21.15.114:0/1912532783 wait complete. 2024-02-20T13:19:24.817 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:24.152026+0000 mgr.smithi038.powepi (mgr.14156) 60 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:24.813885+0000 mon.smithi038 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.114:0/1912532783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:26.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:26.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:26.152437+0000 mgr.smithi038.powepi (mgr.14156) 61 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:28.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 -- 172.21.15.114:0/4158814395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 msgr2=0x7f7c880f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:28.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 --2- 172.21.15.114:0/4158814395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c880f69b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f7c7c004760 tx=0x7f7c7c009ad0).stop 2024-02-20T13:19:28.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 -- 172.21.15.114:0/4158814395 shutdown_connections 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 --2- 172.21.15.114:0/4158814395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c880f69b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 -- 172.21.15.114:0/4158814395 >> 172.21.15.114:0/4158814395 conn(0x7f7c880f1cd0 msgr2=0x7f7c880f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 -- 172.21.15.114:0/4158814395 shutdown_connections 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.225+0000 7f7c8d856700 1 -- 172.21.15.114:0/4158814395 wait complete. 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c8d856700 1 Processor -- start 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c8d856700 1 -- start start 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c8d856700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c881067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c8d856700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88106cb0 con 0x7f7c880f6590 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c86ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c881067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c86ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c881067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43928/0 (socket says 172.21.15.114:43928) 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c86ffd700 1 -- 172.21.15.114:0/1273037481 learned_addr learned my addr 172.21.15.114:0/1273037481 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c86ffd700 1 -- 172.21.15.114:0/1273037481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c009d70 con 0x7f7c880f6590 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.226+0000 7f7c86ffd700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c881067c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f7c7c004170 tx=0x7f7c7c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:28.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.227+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c7c00e040 con 0x7f7c880f6590 2024-02-20T13:19:28.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.227+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c88106eb0 con 0x7f7c880f6590 2024-02-20T13:19:28.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.227+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7c7c008ef0 con 0x7f7c880f6590 2024-02-20T13:19:28.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.227+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c7c014c90 con 0x7f7c880f6590 2024-02-20T13:19:28.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.227+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88107340 con 0x7f7c880f6590 2024-02-20T13:19:28.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.227+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7c7c0283f0 con 0x7f7c880f6590 2024-02-20T13:19:28.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.228+0000 7f7c6f7fe700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c70031090 0x7f7c70033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:28.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.228+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88045bb0 con 0x7f7c880f6590 2024-02-20T13:19:28.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.228+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7c7c042300 con 0x7f7c880f6590 2024-02-20T13:19:28.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.228+0000 7f7c867fc700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c70031090 0x7f7c70033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:28.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.228+0000 7f7c867fc700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c70031090 0x7f7c70033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7c780096a0 tx=0x7f7c78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:28.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.233+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7c7c012d00 con 0x7f7c880f6590 2024-02-20T13:19:28.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.439+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c881081c0 con 0x7f7c880f6590 2024-02-20T13:19:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.440+0000 7f7c6f7fe700 1 -- 172.21.15.114:0/1273037481 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c7c026300 con 0x7f7c880f6590 2024-02-20T13:19:28.441 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:28.442 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c70031090 msgr2=0x7f7c70033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c70031090 0x7f7c70033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7c780096a0 tx=0x7f7c78006b40).stop 2024-02-20T13:19:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 msgr2=0x7f7c881067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c881067c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f7c7c004170 tx=0x7f7c7c009ad0).stop 2024-02-20T13:19:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 shutdown_connections 2024-02-20T13:19:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c70031090 0x7f7c70033550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:28.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 --2- 172.21.15.114:0/1273037481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c880f6590 0x7f7c881067c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:28.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 >> 172.21.15.114:0/1273037481 conn(0x7f7c880f1cd0 msgr2=0x7f7c880f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:28.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.441+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 shutdown_connections 2024-02-20T13:19:28.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:28.442+0000 7f7c8d856700 1 -- 172.21.15.114:0/1273037481 wait complete. 2024-02-20T13:19:28.444 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:28.152680+0000 mgr.smithi038.powepi (mgr.14156) 62 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:28.275301+0000 mon.smithi038 (mon.0) 132 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:19:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:28.276205+0000 mon.smithi038 (mon.0) 133 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:19:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:28.441215+0000 mon.smithi038 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.114:0/1273037481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:30.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:30.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:30.153034+0000 mgr.smithi038.powepi (mgr.14156) 63 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 -- 172.21.15.114:0/1428298033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 msgr2=0x7ff4900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 --2- 172.21.15.114:0/1428298033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff4900f58a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff484009fb0 tx=0x7ff484009b30).stop 2024-02-20T13:19:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 -- 172.21.15.114:0/1428298033 shutdown_connections 2024-02-20T13:19:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 --2- 172.21.15.114:0/1428298033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff4900f58a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 -- 172.21.15.114:0/1428298033 >> 172.21.15.114:0/1428298033 conn(0x7ff4900f0be0 msgr2=0x7ff4900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 -- 172.21.15.114:0/1428298033 shutdown_connections 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.889+0000 7ff494d89700 1 -- 172.21.15.114:0/1428298033 wait complete. 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff494d89700 1 Processor -- start 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff494d89700 1 -- start start 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff494d89700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff49010b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff494d89700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff490107300 con 0x7ff4900f5480 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff48e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff49010b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff48e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff49010b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43944/0 (socket says 172.21.15.114:43944) 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff48e59c700 1 -- 172.21.15.114:0/813872453 learned_addr learned my addr 172.21.15.114:0/813872453 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.890+0000 7ff48e59c700 1 -- 172.21.15.114:0/813872453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484005040 con 0x7ff4900f5480 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff48e59c700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff49010b910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7ff484000c00 tx=0x7ff48400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4840095f0 con 0x7ff4900f5480 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff490060300 con 0x7ff4900f5480 2024-02-20T13:19:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff484007e50 con 0x7ff4900f5480 2024-02-20T13:19:31.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff484014e70 con 0x7ff4900f5480 2024-02-20T13:19:31.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49010be00 con 0x7ff4900f5480 2024-02-20T13:19:31.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.891+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff4840146b0 con 0x7ff4900f5480 2024-02-20T13:19:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.892+0000 7ff47effd700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff478031090 0x7ff478033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.892+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff490045bb0 con 0x7ff4900f5480 2024-02-20T13:19:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.892+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff484041d10 con 0x7ff4900f5480 2024-02-20T13:19:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.892+0000 7ff48dd9b700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff478031090 0x7ff478033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.892+0000 7ff48dd9b700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff478031090 0x7ff478033550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff4800096a0 tx=0x7ff480006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:31.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:31.897+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff48401b4a0 con 0x7ff4900f5480 2024-02-20T13:19:32.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.104+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4900fb060 con 0x7ff4900f5480 2024-02-20T13:19:32.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.104+0000 7ff47effd700 1 -- 172.21.15.114:0/813872453 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff48401b2c0 con 0x7ff4900f5480 2024-02-20T13:19:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:32.106 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:32.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff478031090 msgr2=0x7ff478033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:32.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff478031090 0x7ff478033550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff4800096a0 tx=0x7ff480006b40).stop 2024-02-20T13:19:32.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 msgr2=0x7ff49010b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:32.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff49010b910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7ff484000c00 tx=0x7ff48400b040).stop 2024-02-20T13:19:32.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 shutdown_connections 2024-02-20T13:19:32.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff478031090 0x7ff478033550 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:32.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 --2- 172.21.15.114:0/813872453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4900f5480 0x7ff49010b910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:32.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 >> 172.21.15.114:0/813872453 conn(0x7ff4900f0be0 msgr2=0x7ff4900f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:32.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 shutdown_connections 2024-02-20T13:19:32.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:32.106+0000 7ff494d89700 1 -- 172.21.15.114:0/813872453 wait complete. 2024-02-20T13:19:32.109 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:32.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:32.105891+0000 mon.smithi038 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.114:0/813872453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:33.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:32.153373+0000 mgr.smithi038.powepi (mgr.14156) 64 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:33.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:33.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 -- 172.21.15.114:0/3614196059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 msgr2=0x7f16500f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 --2- 172.21.15.114:0/3614196059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f16500f58d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f1640009fb0 tx=0x7f1640009b30).stop 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 -- 172.21.15.114:0/3614196059 shutdown_connections 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 --2- 172.21.15.114:0/3614196059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f16500f58d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 -- 172.21.15.114:0/3614196059 >> 172.21.15.114:0/3614196059 conn(0x7f16500f0c10 msgr2=0x7f16500f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 -- 172.21.15.114:0/3614196059 shutdown_connections 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.603+0000 7f16580e5700 1 -- 172.21.15.114:0/3614196059 wait complete. 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f16580e5700 1 Processor -- start 2024-02-20T13:19:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f16580e5700 1 -- start start 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f16580e5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f1650105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f16580e5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1650105c10 con 0x7f16500f54b0 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f1655e81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f1650105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f1655e81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f1650105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:49884/0 (socket says 172.21.15.114:49884) 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f1655e81700 1 -- 172.21.15.114:0/3203416353 learned_addr learned my addr 172.21.15.114:0/3203416353 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f1655e81700 1 -- 172.21.15.114:0/3203416353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640005040 con 0x7f16500f54b0 2024-02-20T13:19:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.604+0000 7f1655e81700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f1650105720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f16400089f0 tx=0x7f164000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.605+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640009320 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.605+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1650105e10 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.605+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1640013ec0 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.605+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640014d30 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.605+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16501062a0 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.605+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f16400263a0 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.606+0000 7f16467fc700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f163c031090 0x7f163c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.606+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1640042450 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.606+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16500591e0 con 0x7f16500f54b0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.606+0000 7f1655680700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f163c031090 0x7f163c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.606+0000 7f1655680700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f163c031090 0x7f163c033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f164c0096a0 tx=0x7f164c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.611+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f16400266d0 con 0x7f16500f54b0 2024-02-20T13:19:35.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:34.153644+0000 mgr.smithi038.powepi (mgr.14156) 65 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.817+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16500fafa0 con 0x7f16500f54b0 2024-02-20T13:19:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.818+0000 7f16467fc700 1 -- 172.21.15.114:0/3203416353 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f164002b020 con 0x7f16500f54b0 2024-02-20T13:19:35.820 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:35.820 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:35.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f163c031090 msgr2=0x7f163c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:35.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f163c031090 0x7f163c033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f164c0096a0 tx=0x7f164c006b40).stop 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 msgr2=0x7f1650105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f1650105720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f16400089f0 tx=0x7f164000b040).stop 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 shutdown_connections 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f163c031090 0x7f163c033550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 --2- 172.21.15.114:0/3203416353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16500f54b0 0x7f1650105720 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 >> 172.21.15.114:0/3203416353 conn(0x7f16500f0c10 msgr2=0x7f16500f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 shutdown_connections 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:35.820+0000 7f16580e5700 1 -- 172.21.15.114:0/3203416353 wait complete. 2024-02-20T13:19:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:36.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:35.819762+0000 mon.smithi038 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.114:0/3203416353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:37.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:37.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:37.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:36.153934+0000 mgr.smithi038.powepi (mgr.14156) 66 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:39.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 -- 172.21.15.114:0/1412280484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 msgr2=0x7fc3040f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 --2- 172.21.15.114:0/1412280484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc3040f58d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc2ec009fb0 tx=0x7fc2ec009b30).stop 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 -- 172.21.15.114:0/1412280484 shutdown_connections 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 --2- 172.21.15.114:0/1412280484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc3040f58d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 -- 172.21.15.114:0/1412280484 >> 172.21.15.114:0/1412280484 conn(0x7fc3040f0c10 msgr2=0x7fc3040f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 -- 172.21.15.114:0/1412280484 shutdown_connections 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.357+0000 7fc308d31700 1 -- 172.21.15.114:0/1412280484 wait complete. 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc308d31700 1 Processor -- start 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc308d31700 1 -- start start 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc308d31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc304106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc308d31700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304106d10 con 0x7fc3040f54b0 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc30259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc304106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc30259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc304106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:49912/0 (socket says 172.21.15.114:49912) 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc30259c700 1 -- 172.21.15.114:0/3964827733 learned_addr learned my addr 172.21.15.114:0/3964827733 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:39.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.358+0000 7fc30259c700 1 -- 172.21.15.114:0/3964827733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec005040 con 0x7fc3040f54b0 2024-02-20T13:19:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.359+0000 7fc30259c700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc304106820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc2ec0089f0 tx=0x7fc2ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.359+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2ec009320 con 0x7fc3040f54b0 2024-02-20T13:19:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.359+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc2ec013ec0 con 0x7fc3040f54b0 2024-02-20T13:19:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.359+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc304106f10 con 0x7fc3040f54b0 2024-02-20T13:19:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.359+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2ec014d30 con 0x7fc3040f54b0 2024-02-20T13:19:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.359+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3041073a0 con 0x7fc3040f54b0 2024-02-20T13:19:39.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.360+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc2ec0095f0 con 0x7fc3040f54b0 2024-02-20T13:19:39.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.360+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc304045bb0 con 0x7fc3040f54b0 2024-02-20T13:19:39.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.360+0000 7fc2faffd700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc2f00310e0 0x7fc2f00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:39.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.360+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc2ec042460 con 0x7fc3040f54b0 2024-02-20T13:19:39.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.360+0000 7fc301d9b700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc2f00310e0 0x7fc2f00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:39.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.360+0000 7fc301d9b700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc2f00310e0 0x7fc2f00335a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc2f40096a0 tx=0x7fc2f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:39.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.365+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc2ec01d7f0 con 0x7fc3040f54b0 2024-02-20T13:19:39.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.570+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc304108250 con 0x7fc3040f54b0 2024-02-20T13:19:39.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.570+0000 7fc2faffd700 1 -- 172.21.15.114:0/3964827733 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2ec01d7f0 con 0x7fc3040f54b0 2024-02-20T13:19:39.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:39.572 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc2f00310e0 msgr2=0x7fc2f00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc2f00310e0 0x7fc2f00335a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc2f40096a0 tx=0x7fc2f4006b40).stop 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 msgr2=0x7fc304106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc304106820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc2ec0089f0 tx=0x7fc2ec00b040).stop 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 shutdown_connections 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc2f00310e0 0x7fc2f00335a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 --2- 172.21.15.114:0/3964827733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3040f54b0 0x7fc304106820 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 >> 172.21.15.114:0/3964827733 conn(0x7fc3040f0c10 msgr2=0x7fc30405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:39.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 shutdown_connections 2024-02-20T13:19:39.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:39.572+0000 7fc308d31700 1 -- 172.21.15.114:0/3964827733 wait complete. 2024-02-20T13:19:39.575 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:39.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:38.154257+0000 mgr.smithi038.powepi (mgr.14156) 67 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:40.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:39.572020+0000 mon.smithi038 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.114:0/3964827733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:41.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:41.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:41.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:40.154576+0000 mgr.smithi038.powepi (mgr.14156) 68 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:43.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1924626457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 msgr2=0x7f8e0c0f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 --2- 172.21.15.114:0/1924626457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c0f9010 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8e00009fb0 tx=0x7f8e00009b30).stop 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1924626457 shutdown_connections 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 --2- 172.21.15.114:0/1924626457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c0f9010 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1924626457 >> 172.21.15.114:0/1924626457 conn(0x7f8e0c0f0a30 msgr2=0x7f8e0c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1924626457 shutdown_connections 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1924626457 wait complete. 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.063+0000 7f8e11b9e700 1 Processor -- start 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e11b9e700 1 -- start start 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e11b9e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e11b9e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c106b10 con 0x7f8e0c060300 2024-02-20T13:19:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e10b9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e10b9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:49924/0 (socket says 172.21.15.114:49924) 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e10b9c700 1 -- 172.21.15.114:0/1364424397 learned_addr learned my addr 172.21.15.114:0/1364424397 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e10b9c700 1 -- 172.21.15.114:0/1364424397 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00005040 con 0x7f8e0c060300 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.064+0000 7f8e10b9c700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c106620 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f8e00000c00 tx=0x7f8e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.065+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e000095f0 con 0x7f8e0c060300 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.065+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c106d10 con 0x7f8e0c060300 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.065+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e00007e50 con 0x7f8e0c060300 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.065+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e00014e70 con 0x7f8e0c060300 2024-02-20T13:19:43.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.065+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c1071a0 con 0x7f8e0c060300 2024-02-20T13:19:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.065+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8e00025070 con 0x7f8e0c060300 2024-02-20T13:19:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.066+0000 7f8e097fa700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8dfc031090 0x7f8dfc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.066+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c045bb0 con 0x7f8e0c060300 2024-02-20T13:19:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.066+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e00041d90 con 0x7f8e0c060300 2024-02-20T13:19:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.066+0000 7f8e0bfff700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8dfc031090 0x7f8dfc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.066+0000 7f8e0bfff700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8dfc031090 0x7f8dfc033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f8df80096a0 tx=0x7f8df8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:43.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.071+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e000079c0 con 0x7f8e0c060300 2024-02-20T13:19:43.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.277+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e0c0fbeb0 con 0x7f8e0c060300 2024-02-20T13:19:43.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.278+0000 7f8e097fa700 1 -- 172.21.15.114:0/1364424397 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e0002d090 con 0x7f8e0c060300 2024-02-20T13:19:43.279 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:43.279 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:43.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8dfc031090 msgr2=0x7f8dfc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:43.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8dfc031090 0x7f8dfc033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f8df80096a0 tx=0x7f8df8006b40).stop 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 msgr2=0x7f8e0c106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c106620 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f8e00000c00 tx=0x7f8e0000b040).stop 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 shutdown_connections 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8dfc031090 0x7f8dfc033550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 --2- 172.21.15.114:0/1364424397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e0c060300 0x7f8e0c106620 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 >> 172.21.15.114:0/1364424397 conn(0x7f8e0c0f0a30 msgr2=0x7f8e0c0f1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 shutdown_connections 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:43.280+0000 7f8e11b9e700 1 -- 172.21.15.114:0/1364424397 wait complete. 2024-02-20T13:19:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:43.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:42.154812+0000 mgr.smithi038.powepi (mgr.14156) 69 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:44.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:43.278906+0000 mon.smithi038 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.114:0/1364424397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:45.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:45.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:45.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:44.155150+0000 mgr.smithi038.powepi (mgr.14156) 70 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- 172.21.15.114:0/1934223177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 msgr2=0x7fc9c00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 --2- 172.21.15.114:0/1934223177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c00f56f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc9bc009fb0 tx=0x7fc9bc009b30).stop 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- 172.21.15.114:0/1934223177 shutdown_connections 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 --2- 172.21.15.114:0/1934223177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c00f56f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- 172.21.15.114:0/1934223177 >> 172.21.15.114:0/1934223177 conn(0x7fc9c00f0a30 msgr2=0x7fc9c00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- 172.21.15.114:0/1934223177 shutdown_connections 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- 172.21.15.114:0/1934223177 wait complete. 2024-02-20T13:19:46.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 Processor -- start 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- start start 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c0106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c6f70700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c0106b10 con 0x7fc9c00f52d0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.923+0000 7fc9c5f6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c0106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9c5f6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c0106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37878/0 (socket says 172.21.15.114:37878) 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9c5f6e700 1 -- 172.21.15.114:0/640234800 learned_addr learned my addr 172.21.15.114:0/640234800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9c5f6e700 1 -- 172.21.15.114:0/640234800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc005040 con 0x7fc9c00f52d0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9c5f6e700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c0106620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc9bc000c00 tx=0x7fc9bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9bc0095f0 con 0x7fc9c00f52d0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc9bc007e50 con 0x7fc9c00f52d0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c0106d10 con 0x7fc9c00f52d0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9bc014e70 con 0x7fc9c00f52d0 2024-02-20T13:19:46.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.924+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c01071a0 con 0x7fc9c00f52d0 2024-02-20T13:19:46.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.925+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc9bc025070 con 0x7fc9c00f52d0 2024-02-20T13:19:46.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.925+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c0045bb0 con 0x7fc9c00f52d0 2024-02-20T13:19:46.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.925+0000 7fc9b67fc700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9ac031090 0x7fc9ac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:46.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.925+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc9bc041dd0 con 0x7fc9c00f52d0 2024-02-20T13:19:46.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.925+0000 7fc9c576d700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9ac031090 0x7fc9ac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:46.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.926+0000 7fc9c576d700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9ac031090 0x7fc9ac033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc9b00096a0 tx=0x7fc9b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:46.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:46.930+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc9bc00f4e0 con 0x7fc9c00f52d0 2024-02-20T13:19:47.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.134+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9c0107fa0 con 0x7fc9c00f52d0 2024-02-20T13:19:47.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.135+0000 7fc9b67fc700 1 -- 172.21.15.114:0/640234800 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc9bc00f300 con 0x7fc9c00f52d0 2024-02-20T13:19:47.136 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:47.136 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:47.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9ac031090 msgr2=0x7fc9ac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:47.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9ac031090 0x7fc9ac033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc9b00096a0 tx=0x7fc9b0006b40).stop 2024-02-20T13:19:47.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 msgr2=0x7fc9c0106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c0106620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc9bc000c00 tx=0x7fc9bc00b040).stop 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 shutdown_connections 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc9ac031090 0x7fc9ac033550 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 --2- 172.21.15.114:0/640234800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9c00f52d0 0x7fc9c0106620 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 >> 172.21.15.114:0/640234800 conn(0x7fc9c00f0a30 msgr2=0x7fc9c005ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 shutdown_connections 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:47.136+0000 7fc9c6f70700 1 -- 172.21.15.114:0/640234800 wait complete. 2024-02-20T13:19:47.138 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:47.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:46.155513+0000 mgr.smithi038.powepi (mgr.14156) 71 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:47.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:47.135546+0000 mon.smithi038 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.114:0/640234800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:48.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:48.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:49.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:48.155794+0000 mgr.smithi038.powepi (mgr.14156) 72 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:50.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 -- 172.21.15.114:0/2006860734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 msgr2=0x7f55280f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 --2- 172.21.15.114:0/2006860734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f55280f69b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5518004760 tx=0x7f5518009ad0).stop 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 -- 172.21.15.114:0/2006860734 shutdown_connections 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 --2- 172.21.15.114:0/2006860734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f55280f69b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 -- 172.21.15.114:0/2006860734 >> 172.21.15.114:0/2006860734 conn(0x7f55280f1cd0 msgr2=0x7f55280f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 -- 172.21.15.114:0/2006860734 shutdown_connections 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.795+0000 7f552fa35700 1 -- 172.21.15.114:0/2006860734 wait complete. 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552fa35700 1 Processor -- start 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552fa35700 1 -- start start 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552fa35700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f552806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552fa35700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528069f00 con 0x7f55280f6590 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552d7d1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f552806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:50.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552d7d1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f552806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37900/0 (socket says 172.21.15.114:37900) 2024-02-20T13:19:50.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552d7d1700 1 -- 172.21.15.114:0/2535214061 learned_addr learned my addr 172.21.15.114:0/2535214061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:50.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.796+0000 7f552d7d1700 1 -- 172.21.15.114:0/2535214061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5518009d70 con 0x7f55280f6590 2024-02-20T13:19:50.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.797+0000 7f552d7d1700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f552806ba80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5518004170 tx=0x7f5518009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:50.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.797+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f551800e040 con 0x7f55280f6590 2024-02-20T13:19:50.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.797+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5518008ef0 con 0x7f55280f6590 2024-02-20T13:19:50.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.797+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552806a0a0 con 0x7f55280f6590 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.797+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5518014c90 con 0x7f55280f6590 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.797+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552806a530 con 0x7f55280f6590 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.798+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f55180283f0 con 0x7f55280f6590 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.798+0000 7f551dffb700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f55140310e0 0x7f55140335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.798+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5518042280 con 0x7f55280f6590 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.798+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55280591e0 con 0x7f55280f6590 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.798+0000 7f552cfd0700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f55140310e0 0x7f55140335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.799+0000 7f552cfd0700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f55140310e0 0x7f55140335a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f55240096a0 tx=0x7f5524006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:50.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:50.803+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f55180268f0 con 0x7f55280f6590 2024-02-20T13:19:51.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.010+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f552806ab90 con 0x7f55280f6590 2024-02-20T13:19:51.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.011+0000 7f551dffb700 1 -- 172.21.15.114:0/2535214061 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5518026300 con 0x7f55280f6590 2024-02-20T13:19:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:51.012 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:51.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f55140310e0 msgr2=0x7f55140335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f55140310e0 0x7f55140335a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f55240096a0 tx=0x7f5524006b40).stop 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 msgr2=0x7f552806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f552806ba80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5518004170 tx=0x7f5518009ad0).stop 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 shutdown_connections 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f55140310e0 0x7f55140335a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 --2- 172.21.15.114:0/2535214061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55280f6590 0x7f552806ba80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 >> 172.21.15.114:0/2535214061 conn(0x7f55280f1cd0 msgr2=0x7f55280f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 shutdown_connections 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:51.013+0000 7f552fa35700 1 -- 172.21.15.114:0/2535214061 wait complete. 2024-02-20T13:19:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:51.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:50.156114+0000 mgr.smithi038.powepi (mgr.14156) 73 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:51.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:51.011806+0000 mon.smithi038 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.114:0/2535214061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:52.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:52.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:53.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:52.156493+0000 mgr.smithi038.powepi (mgr.14156) 74 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 -- 172.21.15.114:0/456032320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 msgr2=0x7f25040f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 --2- 172.21.15.114:0/456032320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f25040f58a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f24f4009fb0 tx=0x7f24f4009b30).stop 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 -- 172.21.15.114:0/456032320 shutdown_connections 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 --2- 172.21.15.114:0/456032320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f25040f58a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 -- 172.21.15.114:0/456032320 >> 172.21.15.114:0/456032320 conn(0x7f25040f0be0 msgr2=0x7f25040f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 -- 172.21.15.114:0/456032320 shutdown_connections 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.568+0000 7f250c487700 1 -- 172.21.15.114:0/456032320 wait complete. 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250c487700 1 Processor -- start 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250c487700 1 -- start start 2024-02-20T13:19:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250c487700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f250410b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250c487700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25041072a0 con 0x7f25040f5480 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250a223700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f250410b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250a223700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f250410b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37926/0 (socket says 172.21.15.114:37926) 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.569+0000 7f250a223700 1 -- 172.21.15.114:0/3379576307 learned_addr learned my addr 172.21.15.114:0/3379576307 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f250a223700 1 -- 172.21.15.114:0/3379576307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f4005040 con 0x7f25040f5480 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f250a223700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f250410b8b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f24f40089f0 tx=0x7f24f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24f4009320 con 0x7f25040f5480 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2504060360 con 0x7f25040f5480 2024-02-20T13:19:54.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f24f4013ec0 con 0x7f25040f5480 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24f4014d30 con 0x7f25040f5480 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.570+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250410bda0 con 0x7f25040f5480 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.571+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f24f40095f0 con 0x7f25040f5480 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.571+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2504045bb0 con 0x7f25040f5480 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.571+0000 7f24faffd700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f24f0031090 0x7f24f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.571+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f24f40424a0 con 0x7f25040f5480 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.571+0000 7f2509a22700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f24f0031090 0x7f24f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.572+0000 7f2509a22700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f24f0031090 0x7f24f0033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f25000096a0 tx=0x7f2500006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:54.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.576+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f24f401d7f0 con 0x7f25040f5480 2024-02-20T13:19:54.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.780+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25040faf70 con 0x7f25040f5480 2024-02-20T13:19:54.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.781+0000 7f24faffd700 1 -- 172.21.15.114:0/3379576307 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24f401d7f0 con 0x7f25040f5480 2024-02-20T13:19:54.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:54.782 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:54.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f24f0031090 msgr2=0x7f24f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:54.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f24f0031090 0x7f24f0033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f25000096a0 tx=0x7f2500006b40).stop 2024-02-20T13:19:54.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 msgr2=0x7f250410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f250410b8b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f24f40089f0 tx=0x7f24f400b040).stop 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 shutdown_connections 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f24f0031090 0x7f24f0033550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 --2- 172.21.15.114:0/3379576307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25040f5480 0x7f250410b8b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.782+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 >> 172.21.15.114:0/3379576307 conn(0x7f25040f0be0 msgr2=0x7f25040f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.783+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 shutdown_connections 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:54.783+0000 7f250c487700 1 -- 172.21.15.114:0/3379576307 wait complete. 2024-02-20T13:19:54.784 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:55.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:54.156818+0000 mgr.smithi038.powepi (mgr.14156) 75 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:55.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:54.781452+0000 mon.smithi038 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.114:0/3379576307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:19:56.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:19:56.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:19:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:56.157087+0000 mgr.smithi038.powepi (mgr.14156) 76 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:58.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 -- 172.21.15.114:0/2608716087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 msgr2=0x7f85e00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:58.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 --2- 172.21.15.114:0/2608716087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e00f69b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f85d4009fb0 tx=0x7f85d4009b30).stop 2024-02-20T13:19:58.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 -- 172.21.15.114:0/2608716087 shutdown_connections 2024-02-20T13:19:58.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 --2- 172.21.15.114:0/2608716087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e00f69b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:58.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 -- 172.21.15.114:0/2608716087 >> 172.21.15.114:0/2608716087 conn(0x7f85e00f1cf0 msgr2=0x7f85e00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:58.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 -- 172.21.15.114:0/2608716087 shutdown_connections 2024-02-20T13:19:58.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.369+0000 7f85e5697700 1 -- 172.21.15.114:0/2608716087 wait complete. 2024-02-20T13:19:58.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.370+0000 7f85e5697700 1 Processor -- start 2024-02-20T13:19:58.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.370+0000 7f85e5697700 1 -- start start 2024-02-20T13:19:58.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.370+0000 7f85e5697700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:58.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.370+0000 7f85e5697700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e0106cc0 con 0x7f85e00f6590 2024-02-20T13:19:58.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85deffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:58.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85deffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:59840/0 (socket says 172.21.15.114:59840) 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85deffd700 1 -- 172.21.15.114:0/4030715383 learned_addr learned my addr 172.21.15.114:0/4030715383 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85deffd700 1 -- 172.21.15.114:0/4030715383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d4005040 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85deffd700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e01067d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f85d40089f0 tx=0x7f85d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85d4009320 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e0106ec0 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f85d4013ec0 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85d4014d30 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.371+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e0107350 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.372+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f85d40095f0 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.372+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0045bb0 con 0x7f85e00f6590 2024-02-20T13:19:58.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.372+0000 7f85c77fe700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f85c8031090 0x7f85c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:19:58.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.372+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f85d40423e0 con 0x7f85e00f6590 2024-02-20T13:19:58.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.372+0000 7f85de7fc700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f85c8031090 0x7f85c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:19:58.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.373+0000 7f85de7fc700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f85c8031090 0x7f85c8033550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f85d00096a0 tx=0x7f85d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:19:58.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.377+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f85d401d7f0 con 0x7f85e00f6590 2024-02-20T13:19:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.584+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85e0108200 con 0x7f85e00f6590 2024-02-20T13:19:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.585+0000 7f85c77fe700 1 -- 172.21.15.114:0/4030715383 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85d401d7f0 con 0x7f85e00f6590 2024-02-20T13:19:58.586 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:19:58.586 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:19:58.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f85c8031090 msgr2=0x7f85c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:58.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f85c8031090 0x7f85c8033550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f85d00096a0 tx=0x7f85d0006b40).stop 2024-02-20T13:19:58.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 msgr2=0x7f85e01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:19:58.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e01067d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f85d40089f0 tx=0x7f85d400b040).stop 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 shutdown_connections 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f85c8031090 0x7f85c8033550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 --2- 172.21.15.114:0/4030715383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85e00f6590 0x7f85e01067d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.586+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 >> 172.21.15.114:0/4030715383 conn(0x7f85e00f1cf0 msgr2=0x7f85e00f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.587+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 shutdown_connections 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:19:58.587+0000 7f85e5697700 1 -- 172.21.15.114:0/4030715383 wait complete. 2024-02-20T13:19:58.588 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:19:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:19:58.157404+0000 mgr.smithi038.powepi (mgr.14156) 77 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:19:59.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:19:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:19:58.585731+0000 mon.smithi038 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.114:0/4030715383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:00.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:00.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:00.157687+0000 mgr.smithi038.powepi (mgr.14156) 78 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.323+0000 7fee778c0700 1 -- 172.21.15.114:0/3397942973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee700f6510 msgr2=0x7fee700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.323+0000 7fee778c0700 1 --2- 172.21.15.114:0/3397942973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee700f6510 0x7fee700f6930 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fee60009fb0 tx=0x7fee60009b30).stop 2024-02-20T13:20:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 -- 172.21.15.114:0/3397942973 shutdown_connections 2024-02-20T13:20:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 --2- 172.21.15.114:0/3397942973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee700f6510 0x7fee700f6930 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 -- 172.21.15.114:0/3397942973 >> 172.21.15.114:0/3397942973 conn(0x7fee700f1c70 msgr2=0x7fee700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 -- 172.21.15.114:0/3397942973 shutdown_connections 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 -- 172.21.15.114:0/3397942973 wait complete. 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 Processor -- start 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 -- start start 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 0x7fee70106f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee778c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70107440 con 0x7fee70106b30 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.324+0000 7fee7565c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 0x7fee70106f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee7565c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 0x7fee70106f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:59850/0 (socket says 172.21.15.114:59850) 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee7565c700 1 -- 172.21.15.114:0/2868049421 learned_addr learned my addr 172.21.15.114:0/2868049421 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee7565c700 1 -- 172.21.15.114:0/2868049421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60005040 con 0x7fee70106b30 2024-02-20T13:20:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee7565c700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 0x7fee70106f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fee600134d0 tx=0x7fee60009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee60013ee0 con 0x7fee70106b30 2024-02-20T13:20:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee70107640 con 0x7fee70106b30 2024-02-20T13:20:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fee60013680 con 0x7fee70106b30 2024-02-20T13:20:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee70108190 con 0x7fee70106b30 2024-02-20T13:20:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.325+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee6001ee30 con 0x7fee70106b30 2024-02-20T13:20:02.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.326+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fee60025060 con 0x7fee70106b30 2024-02-20T13:20:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.326+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee700591e0 con 0x7fee70106b30 2024-02-20T13:20:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.326+0000 7fee65ffb700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee5c0310e0 0x7fee5c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.326+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fee60042310 con 0x7fee70106b30 2024-02-20T13:20:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.326+0000 7fee74e5b700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee5c0310e0 0x7fee5c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.327+0000 7fee74e5b700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee5c0310e0 0x7fee5c0335a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fee6c0096a0 tx=0x7fee6c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.331+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fee6001c520 con 0x7fee70106b30 2024-02-20T13:20:02.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.540+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee70045bb0 con 0x7fee70106b30 2024-02-20T13:20:02.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.541+0000 7fee65ffb700 1 -- 172.21.15.114:0/2868049421 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee70045bb0 con 0x7fee70106b30 2024-02-20T13:20:02.542 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:02.542 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee5c0310e0 msgr2=0x7fee5c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee5c0310e0 0x7fee5c0335a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fee6c0096a0 tx=0x7fee6c006b40).stop 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 msgr2=0x7fee70106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 0x7fee70106f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fee600134d0 tx=0x7fee60009b30).stop 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 shutdown_connections 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee5c0310e0 0x7fee5c0335a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 --2- 172.21.15.114:0/2868049421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee70106b30 0x7fee70106f50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:02.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 >> 172.21.15.114:0/2868049421 conn(0x7fee700f1c70 msgr2=0x7fee700f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:02.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 shutdown_connections 2024-02-20T13:20:02.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:02.543+0000 7fee778c0700 1 -- 172.21.15.114:0/2868049421 wait complete. 2024-02-20T13:20:02.545 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:03.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:02.158016+0000 mgr.smithi038.powepi (mgr.14156) 79 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:02.541768+0000 mon.smithi038 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.114:0/2868049421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:04.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:04.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:05.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:04.158408+0000 mgr.smithi038.powepi (mgr.14156) 80 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:06.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 -- 172.21.15.114:0/2440116458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 msgr2=0x7fada00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:06.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 --2- 172.21.15.114:0/2440116458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada00f58a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fad94009fb0 tx=0x7fad94009b30).stop 2024-02-20T13:20:06.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 -- 172.21.15.114:0/2440116458 shutdown_connections 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 --2- 172.21.15.114:0/2440116458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada00f58a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 -- 172.21.15.114:0/2440116458 >> 172.21.15.114:0/2440116458 conn(0x7fada00f0be0 msgr2=0x7fada00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 -- 172.21.15.114:0/2440116458 shutdown_connections 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 -- 172.21.15.114:0/2440116458 wait complete. 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 Processor -- start 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.009+0000 7fada6d00700 1 -- start start 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada6d00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada01094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada6d00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada0060320 con 0x7fada00f5480 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada4a9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada01094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada4a9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada01094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35620/0 (socket says 172.21.15.114:35620) 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada4a9c700 1 -- 172.21.15.114:0/2472303453 learned_addr learned my addr 172.21.15.114:0/2472303453 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada4a9c700 1 -- 172.21.15.114:0/2472303453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94005040 con 0x7fada00f5480 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.010+0000 7fada4a9c700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada01094f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fad940089f0 tx=0x7fad9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:06.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.011+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad94009320 con 0x7fada00f5480 2024-02-20T13:20:06.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.011+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada0060520 con 0x7fada00f5480 2024-02-20T13:20:06.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.011+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad94013ec0 con 0x7fada00f5480 2024-02-20T13:20:06.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.011+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad94014d30 con 0x7fada00f5480 2024-02-20T13:20:06.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.011+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada0109ba0 con 0x7fada00f5480 2024-02-20T13:20:06.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.012+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fad940095f0 con 0x7fada00f5480 2024-02-20T13:20:06.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.012+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada0045bb0 con 0x7fada00f5480 2024-02-20T13:20:06.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.012+0000 7fad9d7fa700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad90031090 0x7fad90033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:06.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.012+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fad940423e0 con 0x7fada00f5480 2024-02-20T13:20:06.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.012+0000 7fad9ffff700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad90031090 0x7fad90033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:06.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.012+0000 7fad9ffff700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad90031090 0x7fad90033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fad8c0096a0 tx=0x7fad8c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:06.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.017+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad9401d7f0 con 0x7fada00f5480 2024-02-20T13:20:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.224+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fada00faf70 con 0x7fada00f5480 2024-02-20T13:20:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.225+0000 7fad9d7fa700 1 -- 172.21.15.114:0/2472303453 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad9401d7f0 con 0x7fada00f5480 2024-02-20T13:20:06.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:06.229 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad90031090 msgr2=0x7fad90033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad90031090 0x7fad90033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fad8c0096a0 tx=0x7fad8c006b40).stop 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 msgr2=0x7fada01094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada01094f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fad940089f0 tx=0x7fad9400b040).stop 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 shutdown_connections 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad90031090 0x7fad90033550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 --2- 172.21.15.114:0/2472303453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada00f5480 0x7fada01094f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 >> 172.21.15.114:0/2472303453 conn(0x7fada00f0be0 msgr2=0x7fada00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 shutdown_connections 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:06.226+0000 7fada6d00700 1 -- 172.21.15.114:0/2472303453 wait complete. 2024-02-20T13:20:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:07.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:06.158748+0000 mgr.smithi038.powepi (mgr.14156) 81 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:07.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:06.226129+0000 mon.smithi038 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.114:0/2472303453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:07.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:07.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:08.159084+0000 mgr.smithi038.powepi (mgr.14156) 82 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 -- 172.21.15.114:0/599975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 msgr2=0x7f283c0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 --2- 172.21.15.114:0/599975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c0f9160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f282c009fb0 tx=0x7f282c009b30).stop 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 -- 172.21.15.114:0/599975425 shutdown_connections 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 --2- 172.21.15.114:0/599975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c0f9160 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 -- 172.21.15.114:0/599975425 >> 172.21.15.114:0/599975425 conn(0x7f283c0f0be0 msgr2=0x7f283c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 -- 172.21.15.114:0/599975425 shutdown_connections 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.794+0000 7f2842a53700 1 -- 172.21.15.114:0/599975425 wait complete. 2024-02-20T13:20:09.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f2842a53700 1 Processor -- start 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f2842a53700 1 -- start start 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f2842a53700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f2842a53700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c106c40 con 0x7f283c060360 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f283bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f283bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35632/0 (socket says 172.21.15.114:35632) 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f283bfff700 1 -- 172.21.15.114:0/3351409526 learned_addr learned my addr 172.21.15.114:0/3351409526 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.795+0000 7f283bfff700 1 -- 172.21.15.114:0/3351409526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c005040 con 0x7f283c060360 2024-02-20T13:20:09.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.796+0000 7f283bfff700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c106750 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f282c009f80 tx=0x7f282c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:09.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.796+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282c009320 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.796+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c106e40 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.796+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f282c013680 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.796+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282c01eda0 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.796+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c1072d0 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.797+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f282c01e420 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.797+0000 7f2838ff9700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2824031090 0x7f2824033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.797+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f282c042360 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.797+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c045bb0 con 0x7f283c060360 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.797+0000 7f283b7fe700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2824031090 0x7f2824033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.797+0000 7f283b7fe700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2824031090 0x7f2824033550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f28300096a0 tx=0x7f2830006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:09.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:09.802+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f282c01e700 con 0x7f283c060360 2024-02-20T13:20:10.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.009+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f283c0fc060 con 0x7f283c060360 2024-02-20T13:20:10.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.009+0000 7f2838ff9700 1 -- 172.21.15.114:0/3351409526 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f282c004490 con 0x7f283c060360 2024-02-20T13:20:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:10.014 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:10.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2824031090 msgr2=0x7f2824033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:10.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2824031090 0x7f2824033550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f28300096a0 tx=0x7f2830006b40).stop 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 msgr2=0x7f283c106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c106750 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f282c009f80 tx=0x7f282c00b040).stop 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 shutdown_connections 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2824031090 0x7f2824033550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 --2- 172.21.15.114:0/3351409526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f283c060360 0x7f283c106750 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 >> 172.21.15.114:0/3351409526 conn(0x7f283c0f0be0 msgr2=0x7f283c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 shutdown_connections 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:10.011+0000 7f2842a53700 1 -- 172.21.15.114:0/3351409526 wait complete. 2024-02-20T13:20:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:10.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:10.011134+0000 mon.smithi038 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.114:0/3351409526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:11.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:11.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:11.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:10.159343+0000 mgr.smithi038.powepi (mgr.14156) 83 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:13.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:12.159687+0000 mgr.smithi038.powepi (mgr.14156) 84 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 -- 172.21.15.114:0/3626513566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 msgr2=0x7fa95c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 --2- 172.21.15.114:0/3626513566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c0f56f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa950009fb0 tx=0x7fa950009b30).stop 2024-02-20T13:20:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 -- 172.21.15.114:0/3626513566 shutdown_connections 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 --2- 172.21.15.114:0/3626513566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c0f56f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 -- 172.21.15.114:0/3626513566 >> 172.21.15.114:0/3626513566 conn(0x7fa95c0f0a30 msgr2=0x7fa95c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 -- 172.21.15.114:0/3626513566 shutdown_connections 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 -- 172.21.15.114:0/3626513566 wait complete. 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 Processor -- start 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.595+0000 7fa961e33700 1 -- start start 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa961e33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa961e33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c060300 con 0x7fa95c0f52d0 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa960e31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa960e31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35660/0 (socket says 172.21.15.114:35660) 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa960e31700 1 -- 172.21.15.114:0/2465330692 learned_addr learned my addr 172.21.15.114:0/2465330692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa960e31700 1 -- 172.21.15.114:0/2465330692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa950005040 con 0x7fa95c0f52d0 2024-02-20T13:20:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.596+0000 7fa960e31700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c109390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa9500089f0 tx=0x7fa95000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:13.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.597+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa950009320 con 0x7fa95c0f52d0 2024-02-20T13:20:13.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.597+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c060500 con 0x7fa95c0f52d0 2024-02-20T13:20:13.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.597+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa950013ec0 con 0x7fa95c0f52d0 2024-02-20T13:20:13.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.597+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa950014d30 con 0x7fa95c0f52d0 2024-02-20T13:20:13.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.597+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c109a40 con 0x7fa95c0f52d0 2024-02-20T13:20:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.597+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa9500263a0 con 0x7fa95c0f52d0 2024-02-20T13:20:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.598+0000 7fa9597fa700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa94c031090 0x7fa94c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.598+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c0591e0 con 0x7fa95c0f52d0 2024-02-20T13:20:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.598+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa950042440 con 0x7fa95c0f52d0 2024-02-20T13:20:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.598+0000 7fa95bfff700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa94c031090 0x7fa94c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.598+0000 7fa95bfff700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa94c031090 0x7fa94c033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa9480096a0 tx=0x7fa948006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:13.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.602+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa950012330 con 0x7fa95c0f52d0 2024-02-20T13:20:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.808+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa95c0fadc0 con 0x7fa95c0f52d0 2024-02-20T13:20:13.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.808+0000 7fa9597fa700 1 -- 172.21.15.114:0/2465330692 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa950014420 con 0x7fa95c0f52d0 2024-02-20T13:20:13.810 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:13.810 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:13.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa94c031090 msgr2=0x7fa94c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:13.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa94c031090 0x7fa94c033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa9480096a0 tx=0x7fa948006b40).stop 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 msgr2=0x7fa95c109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c109390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa9500089f0 tx=0x7fa95000b040).stop 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 shutdown_connections 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa94c031090 0x7fa94c033550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 --2- 172.21.15.114:0/2465330692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa95c0f52d0 0x7fa95c109390 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 >> 172.21.15.114:0/2465330692 conn(0x7fa95c0f0a30 msgr2=0x7fa95c0f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 shutdown_connections 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:13.810+0000 7fa961e33700 1 -- 172.21.15.114:0/2465330692 wait complete. 2024-02-20T13:20:13.812 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:14.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:13.809896+0000 mon.smithi038 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.114:0/2465330692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:15.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:14.160049+0000 mgr.smithi038.powepi (mgr.14156) 85 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:15.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:15.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:17.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:16.160413+0000 mgr.smithi038.powepi (mgr.14156) 86 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 -- 172.21.15.114:0/3799849451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 msgr2=0x7f7c700f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 --2- 172.21.15.114:0/3799849451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c700f69b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f7c58009fb0 tx=0x7f7c58009b30).stop 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 -- 172.21.15.114:0/3799849451 shutdown_connections 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 --2- 172.21.15.114:0/3799849451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c700f69b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 -- 172.21.15.114:0/3799849451 >> 172.21.15.114:0/3799849451 conn(0x7f7c700f1cf0 msgr2=0x7f7c700f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 -- 172.21.15.114:0/3799849451 shutdown_connections 2024-02-20T13:20:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.478+0000 7f7c75285700 1 -- 172.21.15.114:0/3799849451 wait complete. 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c75285700 1 Processor -- start 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c75285700 1 -- start start 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c75285700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c70106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c75285700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70106cf0 con 0x7f7c700f6590 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c6effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c70106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c6effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c70106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44094/0 (socket says 172.21.15.114:44094) 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.479+0000 7f7c6effd700 1 -- 172.21.15.114:0/963488212 learned_addr learned my addr 172.21.15.114:0/963488212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c6effd700 1 -- 172.21.15.114:0/963488212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c58005040 con 0x7f7c700f6590 2024-02-20T13:20:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c6effd700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c70106800 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7c580089f0 tx=0x7f7c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:17.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c58009320 con 0x7f7c700f6590 2024-02-20T13:20:17.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7c58013ec0 con 0x7f7c700f6590 2024-02-20T13:20:17.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c70106ef0 con 0x7f7c700f6590 2024-02-20T13:20:17.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c58014d30 con 0x7f7c700f6590 2024-02-20T13:20:17.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.480+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70107380 con 0x7f7c700f6590 2024-02-20T13:20:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.481+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7c580095f0 con 0x7f7c700f6590 2024-02-20T13:20:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.481+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70045bb0 con 0x7f7c700f6590 2024-02-20T13:20:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.481+0000 7f7c677fe700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c5c0310e0 0x7f7c5c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.481+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7c58042360 con 0x7f7c700f6590 2024-02-20T13:20:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.481+0000 7f7c6e7fc700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c5c0310e0 0x7f7c5c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.482+0000 7f7c6e7fc700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c5c0310e0 0x7f7c5c0335a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f7c600096a0 tx=0x7f7c60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:17.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.486+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7c5801d7f0 con 0x7f7c700f6590 2024-02-20T13:20:17.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.693+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c700fc080 con 0x7f7c700f6590 2024-02-20T13:20:17.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.694+0000 7f7c677fe700 1 -- 172.21.15.114:0/963488212 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c5800fe40 con 0x7f7c700f6590 2024-02-20T13:20:17.696 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:17.696 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c5c0310e0 msgr2=0x7f7c5c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c5c0310e0 0x7f7c5c0335a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f7c600096a0 tx=0x7f7c60006b40).stop 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 msgr2=0x7f7c70106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c70106800 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7c580089f0 tx=0x7f7c5800b040).stop 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 shutdown_connections 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7c5c0310e0 0x7f7c5c0335a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 --2- 172.21.15.114:0/963488212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c700f6590 0x7f7c70106800 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 >> 172.21.15.114:0/963488212 conn(0x7f7c700f1cf0 msgr2=0x7f7c700f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:17.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 shutdown_connections 2024-02-20T13:20:17.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:17.695+0000 7f7c75285700 1 -- 172.21.15.114:0/963488212 wait complete. 2024-02-20T13:20:17.698 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:18.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:17.695711+0000 mon.smithi038 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.114:0/963488212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:19.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:19.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:19.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:18.160714+0000 mgr.smithi038.powepi (mgr.14156) 87 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 -- 172.21.15.114:0/1706317788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 msgr2=0x7feebc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 --2- 172.21.15.114:0/1706317788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc0f69b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feeb0004760 tx=0x7feeb0009ad0).stop 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 -- 172.21.15.114:0/1706317788 shutdown_connections 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 --2- 172.21.15.114:0/1706317788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc0f69b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 -- 172.21.15.114:0/1706317788 >> 172.21.15.114:0/1706317788 conn(0x7feebc0f1cd0 msgr2=0x7feebc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 -- 172.21.15.114:0/1706317788 shutdown_connections 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.168+0000 7feec2463700 1 -- 172.21.15.114:0/1706317788 wait complete. 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feec2463700 1 Processor -- start 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feec2463700 1 -- start start 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feec2463700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:21.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feec2463700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc106cb0 con 0x7feebc0f6590 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feebbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feebbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44110/0 (socket says 172.21.15.114:44110) 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.169+0000 7feebbfff700 1 -- 172.21.15.114:0/457953206 learned_addr learned my addr 172.21.15.114:0/457953206 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feebbfff700 1 -- 172.21.15.114:0/457953206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0009d70 con 0x7feebc0f6590 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feebbfff700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc1067c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7feeb0004170 tx=0x7feeb0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb000e040 con 0x7feebc0f6590 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7feeb0008ef0 con 0x7feebc0f6590 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feebc106eb0 con 0x7feebc0f6590 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb0014c90 con 0x7feebc0f6590 2024-02-20T13:20:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.170+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc107340 con 0x7feebc0f6590 2024-02-20T13:20:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.171+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7feeb0014420 con 0x7feebc0f6590 2024-02-20T13:20:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.171+0000 7feeb8ff9700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feea4031090 0x7feea4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.171+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc045bb0 con 0x7feebc0f6590 2024-02-20T13:20:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.171+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7feeb0042270 con 0x7feebc0f6590 2024-02-20T13:20:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.171+0000 7feebb7fe700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feea4031090 0x7feea4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.172+0000 7feebb7fe700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feea4031090 0x7feea4033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7feeac0096a0 tx=0x7feeac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:21.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.176+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7feeb0026300 con 0x7feebc0f6590 2024-02-20T13:20:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.382+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feebc1081c0 con 0x7feebc0f6590 2024-02-20T13:20:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.383+0000 7feeb8ff9700 1 -- 172.21.15.114:0/457953206 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feeb0012b20 con 0x7feebc0f6590 2024-02-20T13:20:21.385 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:21.385 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feea4031090 msgr2=0x7feea4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feea4031090 0x7feea4033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7feeac0096a0 tx=0x7feeac006b40).stop 2024-02-20T13:20:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 msgr2=0x7feebc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc1067c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7feeb0004170 tx=0x7feeb0009ad0).stop 2024-02-20T13:20:21.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 shutdown_connections 2024-02-20T13:20:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feea4031090 0x7feea4033550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 --2- 172.21.15.114:0/457953206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feebc0f6590 0x7feebc1067c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.384+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 >> 172.21.15.114:0/457953206 conn(0x7feebc0f1cd0 msgr2=0x7feebc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.385+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 shutdown_connections 2024-02-20T13:20:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:21.385+0000 7feec2463700 1 -- 172.21.15.114:0/457953206 wait complete. 2024-02-20T13:20:21.387 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:21.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:20.160992+0000 mgr.smithi038.powepi (mgr.14156) 88 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:22.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:21.384704+0000 mon.smithi038 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.114:0/457953206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:23.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:23.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:23.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:22.161308+0000 mgr.smithi038.powepi (mgr.14156) 89 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 -- 172.21.15.114:0/785357258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 msgr2=0x7f5c700f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 --2- 172.21.15.114:0/785357258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c700f56f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5c58009fb0 tx=0x7f5c58009b30).stop 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 -- 172.21.15.114:0/785357258 shutdown_connections 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 --2- 172.21.15.114:0/785357258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c700f56f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 -- 172.21.15.114:0/785357258 >> 172.21.15.114:0/785357258 conn(0x7f5c700f0a30 msgr2=0x7f5c700f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 -- 172.21.15.114:0/785357258 shutdown_connections 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 -- 172.21.15.114:0/785357258 wait complete. 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 Processor -- start 2024-02-20T13:20:24.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.911+0000 7f5c751b3700 1 -- start start 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c751b3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c70105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c751b3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70105a00 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c70105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c70105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44140/0 (socket says 172.21.15.114:44140) 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c6ffff700 1 -- 172.21.15.114:0/3083069062 learned_addr learned my addr 172.21.15.114:0/3083069062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c6ffff700 1 -- 172.21.15.114:0/3083069062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c58005040 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.912+0000 7f5c6ffff700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c70105510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5c580089f0 tx=0x7f5c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.913+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c58009320 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.913+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5c58013ec0 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.913+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c70105c00 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.913+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c58014d30 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.913+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c70106090 con 0x7f5c700f52d0 2024-02-20T13:20:24.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.914+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5c580095f0 con 0x7f5c700f52d0 2024-02-20T13:20:24.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.914+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c70045bb0 con 0x7f5c700f52d0 2024-02-20T13:20:24.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.914+0000 7f5c6cff9700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5c5c031090 0x7f5c5c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:24.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.914+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5c580423e0 con 0x7f5c700f52d0 2024-02-20T13:20:24.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.914+0000 7f5c6f7fe700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5c5c031090 0x7f5c5c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:24.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.914+0000 7f5c6f7fe700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5c5c031090 0x7f5c5c033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5c600096a0 tx=0x7f5c60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:24.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:24.919+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5c5801d7f0 con 0x7f5c700f52d0 2024-02-20T13:20:25.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.127+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c70106f40 con 0x7f5c700f52d0 2024-02-20T13:20:25.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.127+0000 7f5c6cff9700 1 -- 172.21.15.114:0/3083069062 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c5801d7f0 con 0x7f5c700f52d0 2024-02-20T13:20:25.129 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:25.129 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:25.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5c5c031090 msgr2=0x7f5c5c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:25.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5c5c031090 0x7f5c5c033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5c600096a0 tx=0x7f5c60006b40).stop 2024-02-20T13:20:25.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 msgr2=0x7f5c70105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c70105510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5c580089f0 tx=0x7f5c5800b040).stop 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 shutdown_connections 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5c5c031090 0x7f5c5c033550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 --2- 172.21.15.114:0/3083069062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c700f52d0 0x7f5c70105510 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 >> 172.21.15.114:0/3083069062 conn(0x7f5c700f0a30 msgr2=0x7f5c700f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 shutdown_connections 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:25.129+0000 7f5c751b3700 1 -- 172.21.15.114:0/3083069062 wait complete. 2024-02-20T13:20:25.131 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:24.161619+0000 mgr.smithi038.powepi (mgr.14156) 90 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:25.128932+0000 mon.smithi038 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.114:0/3083069062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:26.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:26.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:26.161966+0000 mgr.smithi038.powepi (mgr.14156) 91 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 -- 172.21.15.114:0/2877926581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 msgr2=0x7f3e1c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 --2- 172.21.15.114:0/2877926581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c0f56f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3e18009fb0 tx=0x7f3e18009b30).stop 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 -- 172.21.15.114:0/2877926581 shutdown_connections 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 --2- 172.21.15.114:0/2877926581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c0f56f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 -- 172.21.15.114:0/2877926581 >> 172.21.15.114:0/2877926581 conn(0x7f3e1c0f0a30 msgr2=0x7f3e1c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 -- 172.21.15.114:0/2877926581 shutdown_connections 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 -- 172.21.15.114:0/2877926581 wait complete. 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 Processor -- start 2024-02-20T13:20:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.712+0000 7f3e220ea700 1 -- start start 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e220ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c105570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e220ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c105a60 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e210e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c105570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e210e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c105570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41996/0 (socket says 172.21.15.114:41996) 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e210e8700 1 -- 172.21.15.114:0/4249751493 learned_addr learned my addr 172.21.15.114:0/4249751493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e210e8700 1 -- 172.21.15.114:0/4249751493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18005040 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.713+0000 7f3e210e8700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c105570 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f3e18000c00 tx=0x7f3e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.714+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e180095f0 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.714+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3e18007e50 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.714+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1c105c60 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.714+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e18014e70 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.714+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1c1060f0 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.715+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3e18025070 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.715+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1c045bb0 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.715+0000 7f3e11ffb700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3e08031090 0x7f3e08033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.715+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3e18041dd0 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.715+0000 7f3e208e7700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3e08031090 0x7f3e08033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.715+0000 7f3e208e7700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3e08031090 0x7f3e08033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3e0c0096a0 tx=0x7f3e0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.720+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3e1800f4e0 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.926+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e1c106ef0 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.927+0000 7f3e11ffb700 1 -- 172.21.15.114:0/4249751493 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e1800f300 con 0x7f3e1c0f52d0 2024-02-20T13:20:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:28.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3e08031090 msgr2=0x7f3e08033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3e08031090 0x7f3e08033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3e0c0096a0 tx=0x7f3e0c006b40).stop 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 msgr2=0x7f3e1c105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c105570 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f3e18000c00 tx=0x7f3e1800b040).stop 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 shutdown_connections 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3e08031090 0x7f3e08033550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 --2- 172.21.15.114:0/4249751493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e1c0f52d0 0x7f3e1c105570 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 >> 172.21.15.114:0/4249751493 conn(0x7f3e1c0f0a30 msgr2=0x7f3e1c0f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 shutdown_connections 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:28.929+0000 7f3e220ea700 1 -- 172.21.15.114:0/4249751493 wait complete. 2024-02-20T13:20:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:28.162213+0000 mgr.smithi038.powepi (mgr.14156) 92 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:28.276515+0000 mon.smithi038 (mon.0) 150 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:20:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:28.277290+0000 mon.smithi038 (mon.0) 151 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:20:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:28.927770+0000 mon.smithi038 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.114:0/4249751493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:30.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:30.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:30.162559+0000 mgr.smithi038.powepi (mgr.14156) 93 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:32.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 -- 172.21.15.114:0/3920551483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 msgr2=0x7f09c00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 --2- 172.21.15.114:0/3920551483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c00f9160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f09b4009fb0 tx=0x7f09b4009b30).stop 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 -- 172.21.15.114:0/3920551483 shutdown_connections 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 --2- 172.21.15.114:0/3920551483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c00f9160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 -- 172.21.15.114:0/3920551483 >> 172.21.15.114:0/3920551483 conn(0x7f09c00f0be0 msgr2=0x7f09c00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 -- 172.21.15.114:0/3920551483 shutdown_connections 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.582+0000 7f09c55b6700 1 -- 172.21.15.114:0/3920551483 wait complete. 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09c55b6700 1 Processor -- start 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09c55b6700 1 -- start start 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09c55b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c0106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09c55b6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0106c40 con 0x7f09c0060360 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09beffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c0106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09beffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c0106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:42008/0 (socket says 172.21.15.114:42008) 2024-02-20T13:20:32.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.583+0000 7f09beffd700 1 -- 172.21.15.114:0/2588132438 learned_addr learned my addr 172.21.15.114:0/2588132438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09beffd700 1 -- 172.21.15.114:0/2588132438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4005040 con 0x7f09c0060360 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09beffd700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c0106750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f09b4009f80 tx=0x7f09b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b4009320 con 0x7f09c0060360 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c0106e40 con 0x7f09c0060360 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f09b4013680 con 0x7f09c0060360 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b401eda0 con 0x7f09c0060360 2024-02-20T13:20:32.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.584+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c01072d0 con 0x7f09c0060360 2024-02-20T13:20:32.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.585+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f09b4014440 con 0x7f09c0060360 2024-02-20T13:20:32.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.585+0000 7f09a77fe700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09a8031090 0x7f09a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:32.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.585+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c0045bb0 con 0x7f09c0060360 2024-02-20T13:20:32.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.585+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f09b4042360 con 0x7f09c0060360 2024-02-20T13:20:32.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.585+0000 7f09be7fc700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09a8031090 0x7f09a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:32.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.586+0000 7f09be7fc700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09a8031090 0x7f09a8033550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f09b00096a0 tx=0x7f09b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:32.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.590+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f09b400fd10 con 0x7f09c0060360 2024-02-20T13:20:32.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.797+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09c01080e0 con 0x7f09c0060360 2024-02-20T13:20:32.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.797+0000 7f09a77fe700 1 -- 172.21.15.114:0/2588132438 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09b4026350 con 0x7f09c0060360 2024-02-20T13:20:32.798 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:32.798 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09a8031090 msgr2=0x7f09a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09a8031090 0x7f09a8033550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f09b00096a0 tx=0x7f09b0006b40).stop 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 msgr2=0x7f09c0106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c0106750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f09b4009f80 tx=0x7f09b400b040).stop 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 shutdown_connections 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09a8031090 0x7f09a8033550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 --2- 172.21.15.114:0/2588132438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09c0060360 0x7f09c0106750 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 >> 172.21.15.114:0/2588132438 conn(0x7f09c00f0be0 msgr2=0x7f09c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 shutdown_connections 2024-02-20T13:20:32.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:32.799+0000 7f09c55b6700 1 -- 172.21.15.114:0/2588132438 wait complete. 2024-02-20T13:20:32.801 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:33.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:32.162941+0000 mgr.smithi038.powepi (mgr.14156) 94 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:32.797886+0000 mon.smithi038 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.114:0/2588132438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:34.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:34.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:34.163258+0000 mgr.smithi038.powepi (mgr.14156) 95 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.250+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3005229332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 msgr2=0x7f5d5c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.250+0000 7f5d60a7e700 1 --2- 172.21.15.114:0/3005229332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c0f58d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5d4c009fb0 tx=0x7f5d4c009b30).stop 2024-02-20T13:20:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.250+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3005229332 shutdown_connections 2024-02-20T13:20:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.250+0000 7f5d60a7e700 1 --2- 172.21.15.114:0/3005229332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c0f58d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.250+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3005229332 >> 172.21.15.114:0/3005229332 conn(0x7f5d5c0f0c10 msgr2=0x7f5d5c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3005229332 shutdown_connections 2024-02-20T13:20:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3005229332 wait complete. 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d60a7e700 1 Processor -- start 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d60a7e700 1 -- start start 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d60a7e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c10b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d60a7e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1072b0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d5a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c10b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d5a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c10b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37998/0 (socket says 172.21.15.114:37998) 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.251+0000 7f5d5a59c700 1 -- 172.21.15.114:0/3502506973 learned_addr learned my addr 172.21.15.114:0/3502506973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d5a59c700 1 -- 172.21.15.114:0/3502506973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c005040 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d5a59c700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c10b8c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5d4c0089f0 tx=0x7f5d4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d4c009320 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c060300 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5d4c013ec0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d4c014d30 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.252+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c10bdb0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.253+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5d4c0095f0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.253+0000 7f5d52ffd700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5d48031090 0x7f5d48033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.253+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c045bb0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.253+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5d4c0424a0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.253+0000 7f5d59d9b700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5d48031090 0x7f5d48033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.254+0000 7f5d59d9b700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5d48031090 0x7f5d48033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5d440096a0 tx=0x7f5d44006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.258+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5d4c01dd70 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.466+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d5c0fafa0 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.467+0000 7f5d52ffd700 1 -- 172.21.15.114:0/3502506973 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d4c02b020 con 0x7f5d5c0f54b0 2024-02-20T13:20:36.468 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:36.468 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:36.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5d48031090 msgr2=0x7f5d48033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5d48031090 0x7f5d48033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5d440096a0 tx=0x7f5d44006b40).stop 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 msgr2=0x7f5d5c10b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c10b8c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5d4c0089f0 tx=0x7f5d4c00b040).stop 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 shutdown_connections 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5d48031090 0x7f5d48033550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 --2- 172.21.15.114:0/3502506973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d5c0f54b0 0x7f5d5c10b8c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 >> 172.21.15.114:0/3502506973 conn(0x7f5d5c0f0c10 msgr2=0x7f5d5c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 shutdown_connections 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:36.469+0000 7f5d60a7e700 1 -- 172.21.15.114:0/3502506973 wait complete. 2024-02-20T13:20:36.470 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:36.163623+0000 mgr.smithi038.powepi (mgr.14156) 96 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:36.467585+0000 mon.smithi038 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.114:0/3502506973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:38.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:38.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:39.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:38.163973+0000 mgr.smithi038.powepi (mgr.14156) 97 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.103+0000 7efd4744c700 1 -- 172.21.15.114:0/2377637350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 msgr2=0x7efd400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.103+0000 7efd4744c700 1 --2- 172.21.15.114:0/2377637350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd400f58a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7efd30009fb0 tx=0x7efd30009b30).stop 2024-02-20T13:20:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.103+0000 7efd4744c700 1 -- 172.21.15.114:0/2377637350 shutdown_connections 2024-02-20T13:20:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.103+0000 7efd4744c700 1 --2- 172.21.15.114:0/2377637350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd400f58a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.103+0000 7efd4744c700 1 -- 172.21.15.114:0/2377637350 >> 172.21.15.114:0/2377637350 conn(0x7efd400f0be0 msgr2=0x7efd400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.104+0000 7efd4744c700 1 -- 172.21.15.114:0/2377637350 shutdown_connections 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.104+0000 7efd4744c700 1 -- 172.21.15.114:0/2377637350 wait complete. 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.104+0000 7efd4744c700 1 Processor -- start 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.104+0000 7efd4744c700 1 -- start start 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.104+0000 7efd4744c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd40109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.104+0000 7efd4744c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd40060300 con 0x7efd400f5480 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd451e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd40109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd451e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd40109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:38022/0 (socket says 172.21.15.114:38022) 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd451e8700 1 -- 172.21.15.114:0/1332422831 learned_addr learned my addr 172.21.15.114:0/1332422831 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd451e8700 1 -- 172.21.15.114:0/1332422831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd30005040 con 0x7efd400f5480 2024-02-20T13:20:40.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd451e8700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd40109510 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7efd300089f0 tx=0x7efd3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:40.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd30009320 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd40060500 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efd30013ec0 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd30014d30 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.105+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd40109bc0 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.106+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efd30014420 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.106+0000 7efd35ffb700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7efd2c031090 0x7efd2c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.106+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efd300424c0 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.106+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd400591e0 con 0x7efd400f5480 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.107+0000 7efd449e7700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7efd2c031090 0x7efd2c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:40.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.107+0000 7efd449e7700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7efd2c031090 0x7efd2c033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7efd3c0096a0 tx=0x7efd3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:40.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.111+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efd30014750 con 0x7efd400f5480 2024-02-20T13:20:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.318+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd40045bb0 con 0x7efd400f5480 2024-02-20T13:20:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.319+0000 7efd35ffb700 1 -- 172.21.15.114:0/1332422831 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd3001d380 con 0x7efd400f5480 2024-02-20T13:20:40.320 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:40.320 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7efd2c031090 msgr2=0x7efd2c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7efd2c031090 0x7efd2c033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7efd3c0096a0 tx=0x7efd3c006b40).stop 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 msgr2=0x7efd40109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd40109510 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7efd300089f0 tx=0x7efd3000b040).stop 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 shutdown_connections 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7efd2c031090 0x7efd2c033550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:40.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 --2- 172.21.15.114:0/1332422831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd400f5480 0x7efd40109510 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:40.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 >> 172.21.15.114:0/1332422831 conn(0x7efd400f0be0 msgr2=0x7efd400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:40.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 shutdown_connections 2024-02-20T13:20:40.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:40.321+0000 7efd4744c700 1 -- 172.21.15.114:0/1332422831 wait complete. 2024-02-20T13:20:40.325 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:40.164269+0000 mgr.smithi038.powepi (mgr.14156) 98 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:40.319342+0000 mon.smithi038 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.114:0/1332422831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:41.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:41.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:43.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:42.164590+0000 mgr.smithi038.powepi (mgr.14156) 99 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:43.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.917+0000 7ff79d99c700 1 -- 172.21.15.114:0/616876629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 msgr2=0x7ff7980f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:43.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.917+0000 7ff79d99c700 1 --2- 172.21.15.114:0/616876629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff7980f56f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ff78c009fb0 tx=0x7ff78c009b30).stop 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.917+0000 7ff79d99c700 1 -- 172.21.15.114:0/616876629 shutdown_connections 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.917+0000 7ff79d99c700 1 --2- 172.21.15.114:0/616876629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff7980f56f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.917+0000 7ff79d99c700 1 -- 172.21.15.114:0/616876629 >> 172.21.15.114:0/616876629 conn(0x7ff7980f0a30 msgr2=0x7ff7980f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.917+0000 7ff79d99c700 1 -- 172.21.15.114:0/616876629 shutdown_connections 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79d99c700 1 -- 172.21.15.114:0/616876629 wait complete. 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79d99c700 1 Processor -- start 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79d99c700 1 -- start start 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79d99c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff798106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79d99c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff798106af0 con 0x7ff7980f52d0 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79c99a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff798106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79c99a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff798106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:38038/0 (socket says 172.21.15.114:38038) 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.918+0000 7ff79c99a700 1 -- 172.21.15.114:0/926975209 learned_addr learned my addr 172.21.15.114:0/926975209 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:43.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff79c99a700 1 -- 172.21.15.114:0/926975209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c005040 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff79c99a700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff798106600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff78c0089f0 tx=0x7ff78c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff78c009320 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff798106cf0 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff78c013ec0 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff78c014d30 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.919+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff798107180 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.920+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff78c0095f0 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.920+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff798045bb0 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.920+0000 7ff7957fa700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff788031090 0x7ff788033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.920+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff78c0424e0 con 0x7ff7980f52d0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.920+0000 7ff797fff700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff788031090 0x7ff788033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.921+0000 7ff797fff700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff788031090 0x7ff788033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff7840096a0 tx=0x7ff784006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:43.925+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff78c01d7f0 con 0x7ff7980f52d0 2024-02-20T13:20:44.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.132+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff798107fd0 con 0x7ff7980f52d0 2024-02-20T13:20:44.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.132+0000 7ff7957fa700 1 -- 172.21.15.114:0/926975209 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff78c00fe40 con 0x7ff7980f52d0 2024-02-20T13:20:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:44.134 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:44.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff788031090 msgr2=0x7ff788033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:44.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff788031090 0x7ff788033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff7840096a0 tx=0x7ff784006b40).stop 2024-02-20T13:20:44.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 msgr2=0x7ff798106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:44.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff798106600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff78c0089f0 tx=0x7ff78c00b040).stop 2024-02-20T13:20:44.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 shutdown_connections 2024-02-20T13:20:44.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff788031090 0x7ff788033550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:44.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 --2- 172.21.15.114:0/926975209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7980f52d0 0x7ff798106600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:44.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 >> 172.21.15.114:0/926975209 conn(0x7ff7980f0a30 msgr2=0x7ff79805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:44.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 shutdown_connections 2024-02-20T13:20:44.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:44.134+0000 7ff79d99c700 1 -- 172.21.15.114:0/926975209 wait complete. 2024-02-20T13:20:44.137 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:44.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:44.133062+0000 mon.smithi038 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.114:0/926975209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:45.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:44.164797+0000 mgr.smithi038.powepi (mgr.14156) 100 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:45.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:45.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:47.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:46.165119+0000 mgr.smithi038.powepi (mgr.14156) 101 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:47.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.911+0000 7f3755b16700 1 -- 172.21.15.114:0/1756467484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 msgr2=0x7f37500f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:47.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.911+0000 7f3755b16700 1 --2- 172.21.15.114:0/1756467484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37500f58a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3740009fb0 tx=0x7f3740009b30).stop 2024-02-20T13:20:47.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.911+0000 7f3755b16700 1 -- 172.21.15.114:0/1756467484 shutdown_connections 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.911+0000 7f3755b16700 1 --2- 172.21.15.114:0/1756467484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37500f58a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.911+0000 7f3755b16700 1 -- 172.21.15.114:0/1756467484 >> 172.21.15.114:0/1756467484 conn(0x7f37500f0be0 msgr2=0x7f37500f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.912+0000 7f3755b16700 1 -- 172.21.15.114:0/1756467484 shutdown_connections 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.912+0000 7f3755b16700 1 -- 172.21.15.114:0/1756467484 wait complete. 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.912+0000 7f3755b16700 1 Processor -- start 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.912+0000 7f3755b16700 1 -- start start 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.912+0000 7f3755b16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37501089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.912+0000 7f3755b16700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3750108ea0 con 0x7f37500f5480 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f374f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37501089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f374f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37501089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58892/0 (socket says 172.21.15.114:58892) 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f374f7fe700 1 -- 172.21.15.114:0/896175104 learned_addr learned my addr 172.21.15.114:0/896175104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f374f7fe700 1 -- 172.21.15.114:0/896175104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3740005040 con 0x7f37500f5480 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f374f7fe700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37501089b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f37400089f0 tx=0x7f374000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3740009320 con 0x7f37500f5480 2024-02-20T13:20:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37501090a0 con 0x7f37500f5480 2024-02-20T13:20:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3740013ec0 con 0x7f37500f5480 2024-02-20T13:20:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3750109530 con 0x7f37500f5480 2024-02-20T13:20:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.913+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3740014d30 con 0x7f37500f5480 2024-02-20T13:20:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.914+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3740014420 con 0x7f37500f5480 2024-02-20T13:20:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.914+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3750045bb0 con 0x7f37500f5480 2024-02-20T13:20:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.914+0000 7f3747fff700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f373c031090 0x7f373c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.914+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f37400423f0 con 0x7f37500f5480 2024-02-20T13:20:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.915+0000 7f374effd700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f373c031090 0x7f373c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.915+0000 7f374effd700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f373c031090 0x7f373c033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f37380096a0 tx=0x7f3738006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:47.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:47.919+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f374001d7f0 con 0x7f37500f5480 2024-02-20T13:20:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.126+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f375010a380 con 0x7f37500f5480 2024-02-20T13:20:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.127+0000 7f3747fff700 1 -- 172.21.15.114:0/896175104 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f374000fe40 con 0x7f37500f5480 2024-02-20T13:20:48.127 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.128+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f373c031090 msgr2=0x7f373c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.128+0000 7f3755b16700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f373c031090 0x7f373c033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f37380096a0 tx=0x7f3738006b40).stop 2024-02-20T13:20:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.128+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 msgr2=0x7f37501089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.128+0000 7f3755b16700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37501089b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f37400089f0 tx=0x7f374000b040).stop 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.129+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 shutdown_connections 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.129+0000 7f3755b16700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f373c031090 0x7f373c033550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.129+0000 7f3755b16700 1 --2- 172.21.15.114:0/896175104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37500f5480 0x7f37501089b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.129+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 >> 172.21.15.114:0/896175104 conn(0x7f37500f0be0 msgr2=0x7f375005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.129+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 shutdown_connections 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:48.129+0000 7f3755b16700 1 -- 172.21.15.114:0/896175104 wait complete. 2024-02-20T13:20:48.130 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:48.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:48.127468+0000 mon.smithi038 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.114:0/896175104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:49.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:48.165511+0000 mgr.smithi038.powepi (mgr.14156) 102 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:49.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:49.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:51.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:50.165849+0000 mgr.smithi038.powepi (mgr.14156) 103 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 -- 172.21.15.114:0/3700009669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 msgr2=0x7f9bb00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 --2- 172.21.15.114:0/3700009669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00f58a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9b98009fb0 tx=0x7f9b98009b30).stop 2024-02-20T13:20:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 -- 172.21.15.114:0/3700009669 shutdown_connections 2024-02-20T13:20:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 --2- 172.21.15.114:0/3700009669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00f58a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 -- 172.21.15.114:0/3700009669 >> 172.21.15.114:0/3700009669 conn(0x7f9bb00f0be0 msgr2=0x7f9bb00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 -- 172.21.15.114:0/3700009669 shutdown_connections 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 -- 172.21.15.114:0/3700009669 wait complete. 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.580+0000 7f9bb4957700 1 Processor -- start 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bb4957700 1 -- start start 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bb4957700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bb4957700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00fb8e0 con 0x7f9bb00f5480 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bae59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bae59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58906/0 (socket says 172.21.15.114:58906) 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bae59c700 1 -- 172.21.15.114:0/1627416166 learned_addr learned my addr 172.21.15.114:0/1627416166 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bae59c700 1 -- 172.21.15.114:0/1627416166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98005040 con 0x7f9bb00f5480 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bae59c700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00fd460 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9b980089f0 tx=0x7f9b9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98009320 con 0x7f9bb00f5480 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb00fba80 con 0x7f9bb00f5480 2024-02-20T13:20:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9b98013ec0 con 0x7f9bb00f5480 2024-02-20T13:20:51.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.581+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98014d30 con 0x7f9bb00f5480 2024-02-20T13:20:51.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.582+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb00fbf10 con 0x7f9bb00f5480 2024-02-20T13:20:51.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.582+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9b980263a0 con 0x7f9bb00f5480 2024-02-20T13:20:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.582+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb00591e0 con 0x7f9bb00f5480 2024-02-20T13:20:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.582+0000 7f9ba6ffd700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b9c031090 0x7f9b9c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.583+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9b98042400 con 0x7f9bb00f5480 2024-02-20T13:20:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.583+0000 7f9badd9b700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b9c031090 0x7f9b9c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.583+0000 7f9badd9b700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b9c031090 0x7f9b9c033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f9ba00096a0 tx=0x7f9ba0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.587+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9b98012330 con 0x7f9bb00f5480 2024-02-20T13:20:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.792+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bb00faf70 con 0x7f9bb00f5480 2024-02-20T13:20:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.793+0000 7f9ba6ffd700 1 -- 172.21.15.114:0/1627416166 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b9801ce90 con 0x7f9bb00f5480 2024-02-20T13:20:51.795 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:51.795 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:51.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b9c031090 msgr2=0x7f9b9c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:51.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b9c031090 0x7f9b9c033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f9ba00096a0 tx=0x7f9ba0006b40).stop 2024-02-20T13:20:51.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 msgr2=0x7f9bb00fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00fd460 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9b980089f0 tx=0x7f9b9800b040).stop 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 shutdown_connections 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b9c031090 0x7f9b9c033550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 --2- 172.21.15.114:0/1627416166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb00f5480 0x7f9bb00fd460 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 >> 172.21.15.114:0/1627416166 conn(0x7f9bb00f0be0 msgr2=0x7f9bb005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 shutdown_connections 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:51.795+0000 7f9bb4957700 1 -- 172.21.15.114:0/1627416166 wait complete. 2024-02-20T13:20:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:52.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:51.794889+0000 mon.smithi038 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.114:0/1627416166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:53.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:52.166183+0000 mgr.smithi038.powepi (mgr.14156) 104 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:53.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:53.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:55.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:54.166459+0000 mgr.smithi038.powepi (mgr.14156) 105 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:55.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 -- 172.21.15.114:0/3642753649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 msgr2=0x7f99e40f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:55.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 --2- 172.21.15.114:0/3642753649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f9200 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f99d4009fb0 tx=0x7f99d4009b30).stop 2024-02-20T13:20:55.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 -- 172.21.15.114:0/3642753649 shutdown_connections 2024-02-20T13:20:55.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 --2- 172.21.15.114:0/3642753649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f9200 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 -- 172.21.15.114:0/3642753649 >> 172.21.15.114:0/3642753649 conn(0x7f99e40f0be0 msgr2=0x7f99e40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 -- 172.21.15.114:0/3642753649 shutdown_connections 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.412+0000 7f99eb64d700 1 -- 172.21.15.114:0/3642753649 wait complete. 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99eb64d700 1 Processor -- start 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99eb64d700 1 -- start start 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99eb64d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99eb64d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40f5100 con 0x7f99e40602c0 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99e93e9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99e93e9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41124/0 (socket says 172.21.15.114:41124) 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99e93e9700 1 -- 172.21.15.114:0/4103149208 learned_addr learned my addr 172.21.15.114:0/4103149208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.413+0000 7f99e93e9700 1 -- 172.21.15.114:0/4103149208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4005040 con 0x7f99e40602c0 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.414+0000 7f99e93e9700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f6c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f99d4013fd0 tx=0x7f99d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.414+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d4018070 con 0x7f99e40602c0 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.414+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e40f52a0 con 0x7f99e40602c0 2024-02-20T13:20:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.414+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f99d4007b10 con 0x7f99e40602c0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.414+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d401c3c0 con 0x7f99e40602c0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.414+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e40f5730 con 0x7f99e40602c0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.415+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f99d401c640 con 0x7f99e40602c0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.415+0000 7f99d9ffb700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f99d0031090 0x7f99d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.415+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f99d4042ab0 con 0x7f99e40602c0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.415+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e40591e0 con 0x7f99e40602c0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.415+0000 7f99e8be8700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f99d0031090 0x7f99d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.415+0000 7f99e8be8700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f99d0031090 0x7f99d0033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f99e00096a0 tx=0x7f99e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:55.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.420+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f99d4024300 con 0x7f99e40602c0 2024-02-20T13:20:55.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.628+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99e40f5d90 con 0x7f99e40602c0 2024-02-20T13:20:55.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.629+0000 7f99d9ffb700 1 -- 172.21.15.114:0/4103149208 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f99d401bb60 con 0x7f99e40602c0 2024-02-20T13:20:55.631 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:55.631 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:55.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f99d0031090 msgr2=0x7f99d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:55.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f99d0031090 0x7f99d0033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f99e00096a0 tx=0x7f99e0006b40).stop 2024-02-20T13:20:55.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 msgr2=0x7f99e40f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:55.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f6c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f99d4013fd0 tx=0x7f99d4009b30).stop 2024-02-20T13:20:55.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 shutdown_connections 2024-02-20T13:20:55.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f99d0031090 0x7f99d0033550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:55.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 --2- 172.21.15.114:0/4103149208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99e40602c0 0x7f99e40f6c80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:55.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 >> 172.21.15.114:0/4103149208 conn(0x7f99e40f0be0 msgr2=0x7f99e40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:55.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.630+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 shutdown_connections 2024-02-20T13:20:55.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:55.631+0000 7f99eb64d700 1 -- 172.21.15.114:0/4103149208 wait complete. 2024-02-20T13:20:55.633 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:56.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:55.630390+0000 mon.smithi038 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.114:0/4103149208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:20:57.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:20:57.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:20:57.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:56.166687+0000 mgr.smithi038.powepi (mgr.14156) 106 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:20:59.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 -- 172.21.15.114:0/3525390599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 msgr2=0x7f48340f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 --2- 172.21.15.114:0/3525390599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f48340f69b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4828004760 tx=0x7f4828009ad0).stop 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 -- 172.21.15.114:0/3525390599 shutdown_connections 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 --2- 172.21.15.114:0/3525390599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f48340f69b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 -- 172.21.15.114:0/3525390599 >> 172.21.15.114:0/3525390599 conn(0x7f48340f1cd0 msgr2=0x7f48340f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 -- 172.21.15.114:0/3525390599 shutdown_connections 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.166+0000 7f4839664700 1 -- 172.21.15.114:0/3525390599 wait complete. 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4839664700 1 Processor -- start 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4839664700 1 -- start start 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4839664700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f483406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4839664700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4834069f00 con 0x7f48340f6590 2024-02-20T13:20:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4832ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f483406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:59.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4832ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f483406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41154/0 (socket says 172.21.15.114:41154) 2024-02-20T13:20:59.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4832ffd700 1 -- 172.21.15.114:0/2381726109 learned_addr learned my addr 172.21.15.114:0/2381726109 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:20:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.167+0000 7f4832ffd700 1 -- 172.21.15.114:0/2381726109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828009d70 con 0x7f48340f6590 2024-02-20T13:20:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.168+0000 7f4832ffd700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f483406ba80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4828004170 tx=0x7f4828009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.168+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f482800e040 con 0x7f48340f6590 2024-02-20T13:20:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.168+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483406a0a0 con 0x7f48340f6590 2024-02-20T13:20:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.168+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4828008ef0 con 0x7f48340f6590 2024-02-20T13:20:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.168+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4828014c90 con 0x7f48340f6590 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.168+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483406a530 con 0x7f48340f6590 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.169+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4828014420 con 0x7f48340f6590 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.169+0000 7f481b7fe700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f481c031090 0x7f481c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.169+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4828042210 con 0x7f48340f6590 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.169+0000 7f48327fc700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f481c031090 0x7f481c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.169+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48340591e0 con 0x7f48340f6590 2024-02-20T13:20:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.169+0000 7f48327fc700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f481c031090 0x7f481c033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f48240096a0 tx=0x7f4824006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:20:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.174+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4828026300 con 0x7f48340f6590 2024-02-20T13:20:59.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.380+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48340fc080 con 0x7f48340f6590 2024-02-20T13:20:59.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.381+0000 7f481b7fe700 1 -- 172.21.15.114:0/2381726109 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4828012b20 con 0x7f48340f6590 2024-02-20T13:20:59.383 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:20:59.383 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f481c031090 msgr2=0x7f481c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f481c031090 0x7f481c033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f48240096a0 tx=0x7f4824006b40).stop 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 msgr2=0x7f483406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f483406ba80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4828004170 tx=0x7f4828009ad0).stop 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 shutdown_connections 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f481c031090 0x7f481c033550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 --2- 172.21.15.114:0/2381726109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48340f6590 0x7f483406ba80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:20:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 >> 172.21.15.114:0/2381726109 conn(0x7f48340f1cd0 msgr2=0x7f48340f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:20:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 shutdown_connections 2024-02-20T13:20:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:20:59.382+0000 7f4839664700 1 -- 172.21.15.114:0/2381726109 wait complete. 2024-02-20T13:20:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:20:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:20:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:20:58.166950+0000 mgr.smithi038.powepi (mgr.14156) 107 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:00.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:20:59.382365+0000 mon.smithi038 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.114:0/2381726109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:01.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:01.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:01.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:00.167214+0000 mgr.smithi038.powepi (mgr.14156) 108 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:02.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 -- 172.21.15.114:0/3023853127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 msgr2=0x7fe8680f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 --2- 172.21.15.114:0/3023853127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8680f69b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fe85c004760 tx=0x7fe85c009ad0).stop 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 -- 172.21.15.114:0/3023853127 shutdown_connections 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 --2- 172.21.15.114:0/3023853127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8680f69b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 -- 172.21.15.114:0/3023853127 >> 172.21.15.114:0/3023853127 conn(0x7fe8680f1cd0 msgr2=0x7fe8680f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 -- 172.21.15.114:0/3023853127 shutdown_connections 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.827+0000 7fe86ef53700 1 -- 172.21.15.114:0/3023853127 wait complete. 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ef53700 1 Processor -- start 2024-02-20T13:21:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ef53700 1 -- start start 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ef53700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8681067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ef53700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe868106cb0 con 0x7fe8680f6590 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ccef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8681067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ccef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8681067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41170/0 (socket says 172.21.15.114:41170) 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.828+0000 7fe86ccef700 1 -- 172.21.15.114:0/758486861 learned_addr learned my addr 172.21.15.114:0/758486861 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe86ccef700 1 -- 172.21.15.114:0/758486861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c009d70 con 0x7fe8680f6590 2024-02-20T13:21:02.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe86ccef700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8681067c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe85c004170 tx=0x7fe85c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe85c00e040 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe868106eb0 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe85c008ef0 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe85c014c90 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.829+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe868107340 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.830+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe85c014420 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.830+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe868045bb0 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.830+0000 7fe8657fa700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe858031090 0x7fe858033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.830+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe85c0422b0 con 0x7fe8680f6590 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.830+0000 7fe867fff700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe858031090 0x7fe858033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:02.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.831+0000 7fe867fff700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe858031090 0x7fe858033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe8540096a0 tx=0x7fe854006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:02.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:02.835+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe85c026300 con 0x7fe8680f6590 2024-02-20T13:21:03.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.041+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8681081f0 con 0x7fe8680f6590 2024-02-20T13:21:03.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe8657fa700 1 -- 172.21.15.114:0/758486861 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe85c012b20 con 0x7fe8680f6590 2024-02-20T13:21:03.044 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:03.044 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe858031090 msgr2=0x7fe858033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe858031090 0x7fe858033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe8540096a0 tx=0x7fe854006b40).stop 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 msgr2=0x7fe8681067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8681067c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe85c004170 tx=0x7fe85c009ad0).stop 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 shutdown_connections 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe858031090 0x7fe858033550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 --2- 172.21.15.114:0/758486861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8680f6590 0x7fe8681067c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:03.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 >> 172.21.15.114:0/758486861 conn(0x7fe8680f1cd0 msgr2=0x7fe8680f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:03.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 shutdown_connections 2024-02-20T13:21:03.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:03.043+0000 7fe86ef53700 1 -- 172.21.15.114:0/758486861 wait complete. 2024-02-20T13:21:03.046 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:02.167409+0000 mgr.smithi038.powepi (mgr.14156) 109 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:03.043533+0000 mon.smithi038 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.114:0/758486861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:04.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:04.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:05.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:04.167744+0000 mgr.smithi038.powepi (mgr.14156) 110 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:06.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 -- 172.21.15.114:0/3717406661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 msgr2=0x7f08a00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:06.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 --2- 172.21.15.114:0/3717406661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a00f9160 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0894009fb0 tx=0x7f0894009b30).stop 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 -- 172.21.15.114:0/3717406661 shutdown_connections 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 --2- 172.21.15.114:0/3717406661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a00f9160 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 -- 172.21.15.114:0/3717406661 >> 172.21.15.114:0/3717406661 conn(0x7f08a00f0be0 msgr2=0x7f08a00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 -- 172.21.15.114:0/3717406661 shutdown_connections 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 -- 172.21.15.114:0/3717406661 wait complete. 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 Processor -- start 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.606+0000 7f08a4ba2700 1 -- start start 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f08a4ba2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a0106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f08a4ba2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a0106c60 con 0x7f08a0060360 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f089e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a0106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f089e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a0106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34046/0 (socket says 172.21.15.114:34046) 2024-02-20T13:21:06.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f089e59c700 1 -- 172.21.15.114:0/1032484009 learned_addr learned my addr 172.21.15.114:0/1032484009 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f089e59c700 1 -- 172.21.15.114:0/1032484009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894005040 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f089e59c700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a0106770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f0894009f80 tx=0x7f089400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0894009320 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a0106e60 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0894013680 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f089401eda0 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.607+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a01072f0 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.608+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f089401e420 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.608+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a0045bb0 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.608+0000 7f088effd700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0888031090 0x7f0888033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.608+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0894042470 con 0x7f08a0060360 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.609+0000 7f089dd9b700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0888031090 0x7f0888033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.609+0000 7f089dd9b700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0888031090 0x7f0888033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f08900096a0 tx=0x7f0890006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:06.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.613+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f089401d380 con 0x7f08a0060360 2024-02-20T13:21:06.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.819+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08a01081a0 con 0x7f08a0060360 2024-02-20T13:21:06.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.820+0000 7f088effd700 1 -- 172.21.15.114:0/1032484009 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0894004490 con 0x7f08a0060360 2024-02-20T13:21:06.822 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:06.822 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0888031090 msgr2=0x7f0888033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0888031090 0x7f0888033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f08900096a0 tx=0x7f0890006b40).stop 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 msgr2=0x7f08a0106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a0106770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f0894009f80 tx=0x7f089400b040).stop 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 shutdown_connections 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0888031090 0x7f0888033550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 --2- 172.21.15.114:0/1032484009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08a0060360 0x7f08a0106770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 >> 172.21.15.114:0/1032484009 conn(0x7f08a00f0be0 msgr2=0x7f08a00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 shutdown_connections 2024-02-20T13:21:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:06.821+0000 7f08a4ba2700 1 -- 172.21.15.114:0/1032484009 wait complete. 2024-02-20T13:21:06.824 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:07.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:06.168094+0000 mgr.smithi038.powepi (mgr.14156) 111 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:07.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:06.821160+0000 mon.smithi038 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.114:0/1032484009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:08.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:08.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:09.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:08.168441+0000 mgr.smithi038.powepi (mgr.14156) 112 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/3390355719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 msgr2=0x7f2ccc0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 --2- 172.21.15.114:0/3390355719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc0f58d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f2cc0009fb0 tx=0x7f2cc0009b30).stop 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/3390355719 shutdown_connections 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 --2- 172.21.15.114:0/3390355719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc0f58d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/3390355719 >> 172.21.15.114:0/3390355719 conn(0x7f2ccc0f0c10 msgr2=0x7f2ccc0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/3390355719 shutdown_connections 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.316+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/3390355719 wait complete. 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cd0b0e700 1 Processor -- start 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cd0b0e700 1 -- start start 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cd0b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc1094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:10.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cd0b0e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc060300 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cca59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc1094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cca59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc1094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34064/0 (socket says 172.21.15.114:34064) 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cca59c700 1 -- 172.21.15.114:0/2275420542 learned_addr learned my addr 172.21.15.114:0/2275420542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.317+0000 7f2cca59c700 1 -- 172.21.15.114:0/2275420542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0005040 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cca59c700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc1094c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f2cc00089f0 tx=0x7f2cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cc0009320 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ccc060500 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2cc0013ec0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cc0014d30 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc109b70 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.318+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2cc00095f0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.319+0000 7f2cbaffd700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2cb4031090 0x7f2cb4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.319+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2cc00424a0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.319+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc0591e0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.319+0000 7f2cc9d9b700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2cb4031090 0x7f2cb4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:10.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.319+0000 7f2cc9d9b700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2cb4031090 0x7f2cb4033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2cbc0096a0 tx=0x7f2cbc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:10.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.324+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2cc001d7f0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.532+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ccc045bb0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.533+0000 7f2cbaffd700 1 -- 172.21.15.114:0/2275420542 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2cc001d7f0 con 0x7f2ccc0f54b0 2024-02-20T13:21:10.535 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:10.535 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2cb4031090 msgr2=0x7f2cb4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2cb4031090 0x7f2cb4033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2cbc0096a0 tx=0x7f2cbc006b40).stop 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 msgr2=0x7f2ccc1094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc1094c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f2cc00089f0 tx=0x7f2cc000b040).stop 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 shutdown_connections 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2cb4031090 0x7f2cb4033550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 --2- 172.21.15.114:0/2275420542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ccc0f54b0 0x7f2ccc1094c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 >> 172.21.15.114:0/2275420542 conn(0x7f2ccc0f0c10 msgr2=0x7f2ccc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 shutdown_connections 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:10.535+0000 7f2cd0b0e700 1 -- 172.21.15.114:0/2275420542 wait complete. 2024-02-20T13:21:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:11.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:10.168823+0000 mgr.smithi038.powepi (mgr.14156) 113 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:11.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:10.534779+0000 mon.smithi038 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.114:0/2275420542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:12.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:12.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:13.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:12.169048+0000 mgr.smithi038.powepi (mgr.14156) 114 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- 172.21.15.114:0/853368867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 msgr2=0x7f89200f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 --2- 172.21.15.114:0/853368867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f89200f56f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8914009fb0 tx=0x7f8914009b30).stop 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- 172.21.15.114:0/853368867 shutdown_connections 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 --2- 172.21.15.114:0/853368867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f89200f56f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- 172.21.15.114:0/853368867 >> 172.21.15.114:0/853368867 conn(0x7f89200f0a30 msgr2=0x7f89200f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- 172.21.15.114:0/853368867 shutdown_connections 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- 172.21.15.114:0/853368867 wait complete. 2024-02-20T13:21:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 Processor -- start 2024-02-20T13:21:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- start start 2024-02-20T13:21:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f892010b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:14.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.147+0000 7f8925e59700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8920107120 con 0x7f89200f52d0 2024-02-20T13:21:14.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8924e57700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f892010b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:14.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8924e57700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f892010b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34080/0 (socket says 172.21.15.114:34080) 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8924e57700 1 -- 172.21.15.114:0/2506357219 learned_addr learned my addr 172.21.15.114:0/2506357219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8924e57700 1 -- 172.21.15.114:0/2506357219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8914005040 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8924e57700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f892010b730 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f89140089f0 tx=0x7f891400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8914009320 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8920060300 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8914013ec0 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8914014d30 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.148+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892010bc20 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.149+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f89140263a0 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.149+0000 7f891d7fa700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8910031090 0x7f8910033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.149+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f89140423d0 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.149+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89200591e0 con 0x7f89200f52d0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.150+0000 7f891ffff700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8910031090 0x7f8910033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:14.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.150+0000 7f891ffff700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8910031090 0x7f8910033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f890c0096a0 tx=0x7f890c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:14.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.154+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8914012920 con 0x7f89200f52d0 2024-02-20T13:21:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.358+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8920045bb0 con 0x7f89200f52d0 2024-02-20T13:21:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.359+0000 7f891d7fa700 1 -- 172.21.15.114:0/2506357219 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8914012330 con 0x7f89200f52d0 2024-02-20T13:21:14.359 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:14.360 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8910031090 msgr2=0x7f8910033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8910031090 0x7f8910033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f890c0096a0 tx=0x7f890c006b40).stop 2024-02-20T13:21:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 msgr2=0x7f892010b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f892010b730 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f89140089f0 tx=0x7f891400b040).stop 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 shutdown_connections 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8910031090 0x7f8910033550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 --2- 172.21.15.114:0/2506357219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89200f52d0 0x7f892010b730 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 >> 172.21.15.114:0/2506357219 conn(0x7f89200f0a30 msgr2=0x7f89200f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 shutdown_connections 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:14.360+0000 7f8925e59700 1 -- 172.21.15.114:0/2506357219 wait complete. 2024-02-20T13:21:14.362 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:15.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:14.169406+0000 mgr.smithi038.powepi (mgr.14156) 115 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:15.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:14.359546+0000 mon.smithi038 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.114:0/2506357219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:15.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:15.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:17.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:16.169795+0000 mgr.smithi038.powepi (mgr.14156) 116 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 -- 172.21.15.114:0/1221475826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 msgr2=0x7fb3640f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 --2- 172.21.15.114:0/1221475826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb3640f58a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb34c009fb0 tx=0x7fb34c009b30).stop 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 -- 172.21.15.114:0/1221475826 shutdown_connections 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 --2- 172.21.15.114:0/1221475826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb3640f58a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 -- 172.21.15.114:0/1221475826 >> 172.21.15.114:0/1221475826 conn(0x7fb3640f0be0 msgr2=0x7fb3640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 -- 172.21.15.114:0/1221475826 shutdown_connections 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.764+0000 7fb369b07700 1 -- 172.21.15.114:0/1221475826 wait complete. 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb369b07700 1 Processor -- start 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb369b07700 1 -- start start 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb369b07700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb364109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb369b07700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364060300 con 0x7fb3640f5480 2024-02-20T13:21:17.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb3637fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb364109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb3637fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb364109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47868/0 (socket says 172.21.15.114:47868) 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.765+0000 7fb3637fe700 1 -- 172.21.15.114:0/3322079259 learned_addr learned my addr 172.21.15.114:0/3322079259 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb3637fe700 1 -- 172.21.15.114:0/3322079259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c005040 con 0x7fb3640f5480 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb3637fe700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb364109510 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fb34c0089f0 tx=0x7fb34c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb34c009320 con 0x7fb3640f5480 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364060500 con 0x7fb3640f5480 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb34c013ec0 con 0x7fb3640f5480 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb34c014d30 con 0x7fb3640f5480 2024-02-20T13:21:17.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.766+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb364109bc0 con 0x7fb3640f5480 2024-02-20T13:21:17.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.767+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb34c0095f0 con 0x7fb3640f5480 2024-02-20T13:21:17.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.767+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3640591e0 con 0x7fb3640f5480 2024-02-20T13:21:17.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.767+0000 7fb35bfff700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb3500310e0 0x7fb3500335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:17.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.767+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb34c0424a0 con 0x7fb3640f5480 2024-02-20T13:21:17.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.767+0000 7fb362ffd700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb3500310e0 0x7fb3500335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:17.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.768+0000 7fb362ffd700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb3500310e0 0x7fb3500335a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:17.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.772+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb34c01d980 con 0x7fb3640f5480 2024-02-20T13:21:17.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.977+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb364045bb0 con 0x7fb3640f5480 2024-02-20T13:21:17.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.978+0000 7fb35bfff700 1 -- 172.21.15.114:0/3322079259 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb34c01d380 con 0x7fb3640f5480 2024-02-20T13:21:17.979 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:17.979 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:17.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb3500310e0 msgr2=0x7fb3500335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb3500310e0 0x7fb3500335a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).stop 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 msgr2=0x7fb364109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb364109510 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fb34c0089f0 tx=0x7fb34c00b040).stop 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 shutdown_connections 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb3500310e0 0x7fb3500335a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 --2- 172.21.15.114:0/3322079259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3640f5480 0x7fb364109510 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 >> 172.21.15.114:0/3322079259 conn(0x7fb3640f0be0 msgr2=0x7fb3640f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 shutdown_connections 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:17.980+0000 7fb369b07700 1 -- 172.21.15.114:0/3322079259 wait complete. 2024-02-20T13:21:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:18.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:17.978584+0000 mon.smithi038 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.114:0/3322079259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:19.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:19.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:19.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:18.170142+0000 mgr.smithi038.powepi (mgr.14156) 117 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.605+0000 7ff339f1b700 1 -- 172.21.15.114:0/1670351836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 msgr2=0x7ff3340f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 --2- 172.21.15.114:0/1670351836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff3340f56f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7ff32c009960 tx=0x7ff32c004d10).stop 2024-02-20T13:21:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 -- 172.21.15.114:0/1670351836 shutdown_connections 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 --2- 172.21.15.114:0/1670351836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff3340f56f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 -- 172.21.15.114:0/1670351836 >> 172.21.15.114:0/1670351836 conn(0x7ff3340f0a30 msgr2=0x7ff3340f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 -- 172.21.15.114:0/1670351836 shutdown_connections 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 -- 172.21.15.114:0/1670351836 wait complete. 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 Processor -- start 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 -- start start 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff334109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.606+0000 7ff339f1b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334060320 con 0x7ff3340f52d0 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff338f19700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff334109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff338f19700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff334109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47888/0 (socket says 172.21.15.114:47888) 2024-02-20T13:21:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff338f19700 1 -- 172.21.15.114:0/4137445990 learned_addr learned my addr 172.21.15.114:0/4137445990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff338f19700 1 -- 172.21.15.114:0/4137445990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c009640 con 0x7ff3340f52d0 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff338f19700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff334109310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7ff32c009610 tx=0x7ff32c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff32c018070 con 0x7ff3340f52d0 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334060520 con 0x7ff3340f52d0 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff32c00b7e0 con 0x7ff3340f52d0 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff32c01d420 con 0x7ff3340f52d0 2024-02-20T13:21:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.607+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3341099c0 con 0x7ff3340f52d0 2024-02-20T13:21:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.608+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff32c024060 con 0x7ff3340f52d0 2024-02-20T13:21:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.608+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334045bb0 con 0x7ff3340f52d0 2024-02-20T13:21:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.608+0000 7ff3317fa700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff31c031090 0x7ff31c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.608+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff32c042630 con 0x7ff3340f52d0 2024-02-20T13:21:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.608+0000 7ff333fff700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff31c031090 0x7ff31c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.609+0000 7ff333fff700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff31c031090 0x7ff31c033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff3280096a0 tx=0x7ff328006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.613+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff32c0048d0 con 0x7ff3340f52d0 2024-02-20T13:21:21.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:20.170474+0000 mgr.smithi038.powepi (mgr.14156) 118 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.818+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff33410a1e0 con 0x7ff3340f52d0 2024-02-20T13:21:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.819+0000 7ff3317fa700 1 -- 172.21.15.114:0/4137445990 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff32c0048d0 con 0x7ff3340f52d0 2024-02-20T13:21:21.820 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:21.820 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:21.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff31c031090 msgr2=0x7ff31c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:21.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff31c031090 0x7ff31c033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff3280096a0 tx=0x7ff328006b40).stop 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 msgr2=0x7ff334109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff334109310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7ff32c009610 tx=0x7ff32c004d10).stop 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 shutdown_connections 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff31c031090 0x7ff31c033550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 --2- 172.21.15.114:0/4137445990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3340f52d0 0x7ff334109310 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 >> 172.21.15.114:0/4137445990 conn(0x7ff3340f0a30 msgr2=0x7ff3340f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 shutdown_connections 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:21.821+0000 7ff339f1b700 1 -- 172.21.15.114:0/4137445990 wait complete. 2024-02-20T13:21:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:22.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:21.819545+0000 mon.smithi038 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.114:0/4137445990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:23.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:23.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:23.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:22.170812+0000 mgr.smithi038.powepi (mgr.14156) 119 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:25.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 -- 172.21.15.114:0/2197693406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc0f4850 msgr2=0x7fafbc0f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:25.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 --2- 172.21.15.114:0/2197693406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc0f4850 0x7fafbc0f4c70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fafac009fb0 tx=0x7fafac009b30).stop 2024-02-20T13:21:25.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 -- 172.21.15.114:0/2197693406 shutdown_connections 2024-02-20T13:21:25.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 --2- 172.21.15.114:0/2197693406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc0f4850 0x7fafbc0f4c70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:25.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 -- 172.21.15.114:0/2197693406 >> 172.21.15.114:0/2197693406 conn(0x7fafbc0f0390 msgr2=0x7fafbc0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 -- 172.21.15.114:0/2197693406 shutdown_connections 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 -- 172.21.15.114:0/2197693406 wait complete. 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.336+0000 7fafc2900700 1 Processor -- start 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafc2900700 1 -- start start 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafc2900700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 0x7fafbc106ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafc2900700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1071d0 con 0x7fafbc1068c0 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafbbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 0x7fafbc106ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafbbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 0x7fafbc106ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43634/0 (socket says 172.21.15.114:43634) 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafbbfff700 1 -- 172.21.15.114:0/1846221764 learned_addr learned my addr 172.21.15.114:0/1846221764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.337+0000 7fafbbfff700 1 -- 172.21.15.114:0/1846221764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac005040 con 0x7fafbc1068c0 2024-02-20T13:21:25.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.338+0000 7fafbbfff700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 0x7fafbc106ce0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fafac009880 tx=0x7fafac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.338+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafac013680 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.338+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc1073d0 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.338+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fafac0095f0 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.338+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafac014e40 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.338+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc107f20 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.339+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fafac014420 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.339+0000 7fafb8ff9700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fafa8031090 0x7fafa8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.339+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fafac042480 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.339+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc0591e0 con 0x7fafbc1068c0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.339+0000 7fafbb7fe700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fafa8031090 0x7fafa8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:25.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.340+0000 7fafbb7fe700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fafa8031090 0x7fafa8033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fafa40096a0 tx=0x7fafa4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:25.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.344+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fafac025ae0 con 0x7fafbc1068c0 2024-02-20T13:21:25.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.550+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafbc1085b0 con 0x7fafbc1068c0 2024-02-20T13:21:25.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.550+0000 7fafb8ff9700 1 -- 172.21.15.114:0/1846221764 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fafac025ae0 con 0x7fafbc1068c0 2024-02-20T13:21:25.551 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:25.551 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:25.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fafa8031090 msgr2=0x7fafa8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fafa8031090 0x7fafa8033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fafa40096a0 tx=0x7fafa4006b40).stop 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 msgr2=0x7fafbc106ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 0x7fafbc106ce0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fafac009880 tx=0x7fafac009b30).stop 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 shutdown_connections 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fafa8031090 0x7fafa8033550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 --2- 172.21.15.114:0/1846221764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafbc1068c0 0x7fafbc106ce0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 >> 172.21.15.114:0/1846221764 conn(0x7fafbc0f0390 msgr2=0x7fafbc0f0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 shutdown_connections 2024-02-20T13:21:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:25.552+0000 7fafc2900700 1 -- 172.21.15.114:0/1846221764 wait complete. 2024-02-20T13:21:25.554 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:24.171138+0000 mgr.smithi038.powepi (mgr.14156) 120 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:26.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:26 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:25.550909+0000 mon.smithi038 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.114:0/1846221764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:27.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:27.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:27.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:26.171531+0000 mgr.smithi038.powepi (mgr.14156) 121 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:29.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 -- 172.21.15.114:0/3914574690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 msgr2=0x7fe4b80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 --2- 172.21.15.114:0/3914574690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80f56f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fe4a8009fb0 tx=0x7fe4a8009b30).stop 2024-02-20T13:21:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 -- 172.21.15.114:0/3914574690 shutdown_connections 2024-02-20T13:21:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 --2- 172.21.15.114:0/3914574690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80f56f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 -- 172.21.15.114:0/3914574690 >> 172.21.15.114:0/3914574690 conn(0x7fe4b80f0a30 msgr2=0x7fe4b80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 -- 172.21.15.114:0/3914574690 shutdown_connections 2024-02-20T13:21:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 -- 172.21.15.114:0/3914574690 wait complete. 2024-02-20T13:21:29.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 Processor -- start 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.115+0000 7fe4b7fff700 1 -- start start 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.116+0000 7fe4b7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.116+0000 7fe4b7fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b80fb730 con 0x7fe4b80f52d0 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.116+0000 7fe4b6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.116+0000 7fe4b6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43650/0 (socket says 172.21.15.114:43650) 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.116+0000 7fe4b6ffd700 1 -- 172.21.15.114:0/2670160062 learned_addr learned my addr 172.21.15.114:0/2670160062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.116+0000 7fe4b6ffd700 1 -- 172.21.15.114:0/2670160062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a8005040 con 0x7fe4b80f52d0 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.117+0000 7fe4b6ffd700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80fd2b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fe4a80089f0 tx=0x7fe4a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.117+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a8009320 con 0x7fe4b80f52d0 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.117+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b80fb8d0 con 0x7fe4b80f52d0 2024-02-20T13:21:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.117+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b80fbd60 con 0x7fe4b80f52d0 2024-02-20T13:21:29.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.117+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe4a8013ec0 con 0x7fe4b80f52d0 2024-02-20T13:21:29.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.117+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a8014d00 con 0x7fe4b80f52d0 2024-02-20T13:21:29.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.118+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe4a80263a0 con 0x7fe4b80f52d0 2024-02-20T13:21:29.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.118+0000 7fe49f7fe700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe4a0031090 0x7fe4a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:29.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.118+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe4a80424c0 con 0x7fe4b80f52d0 2024-02-20T13:21:29.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.118+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b80591e0 con 0x7fe4b80f52d0 2024-02-20T13:21:29.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.118+0000 7fe4b67fc700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe4a0031090 0x7fe4a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:29.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.118+0000 7fe4b67fc700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe4a0031090 0x7fe4a0033550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe4ac0096a0 tx=0x7fe4ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:29.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.123+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe4a8012920 con 0x7fe4b80f52d0 2024-02-20T13:21:29.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.327+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4b80fc460 con 0x7fe4b80f52d0 2024-02-20T13:21:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.328+0000 7fe49f7fe700 1 -- 172.21.15.114:0/2670160062 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe4a802b020 con 0x7fe4b80f52d0 2024-02-20T13:21:29.329 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:29.329 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe4a0031090 msgr2=0x7fe4a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe4a0031090 0x7fe4a0033550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe4ac0096a0 tx=0x7fe4ac006b40).stop 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 msgr2=0x7fe4b80fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80fd2b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fe4a80089f0 tx=0x7fe4a800b040).stop 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 shutdown_connections 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe4a0031090 0x7fe4a0033550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 --2- 172.21.15.114:0/2670160062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4b80f52d0 0x7fe4b80fd2b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 >> 172.21.15.114:0/2670160062 conn(0x7fe4b80f0a30 msgr2=0x7fe4b80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 shutdown_connections 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:29.330+0000 7fe4b7fff700 1 -- 172.21.15.114:0/2670160062 wait complete. 2024-02-20T13:21:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:29.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:28.171835+0000 mgr.smithi038.powepi (mgr.14156) 122 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:28.277707+0000 mon.smithi038 (mon.0) 168 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:21:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:28.278106+0000 mon.smithi038 (mon.0) 169 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:21:30.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:30 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:29.328820+0000 mon.smithi038 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.114:0/2670160062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:30.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:30.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:30.172156+0000 mgr.smithi038.powepi (mgr.14156) 123 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:32.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- 172.21.15.114:0/2677240979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 msgr2=0x7ffb7c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:32.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 --2- 172.21.15.114:0/2677240979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c0f58a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7ffb6c009fb0 tx=0x7ffb6c009b30).stop 2024-02-20T13:21:32.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- 172.21.15.114:0/2677240979 shutdown_connections 2024-02-20T13:21:32.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 --2- 172.21.15.114:0/2677240979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c0f58a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:32.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- 172.21.15.114:0/2677240979 >> 172.21.15.114:0/2677240979 conn(0x7ffb7c0f0be0 msgr2=0x7ffb7c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- 172.21.15.114:0/2677240979 shutdown_connections 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- 172.21.15.114:0/2677240979 wait complete. 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 Processor -- start 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- start start 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.852+0000 7ffb808b1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c106cd0 con 0x7ffb7c0f5480 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb7a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb7a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43668/0 (socket says 172.21.15.114:43668) 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb7a59c700 1 -- 172.21.15.114:0/567419344 learned_addr learned my addr 172.21.15.114:0/567419344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb7a59c700 1 -- 172.21.15.114:0/567419344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c005040 con 0x7ffb7c0f5480 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb7a59c700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c1067e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7ffb6c0089f0 tx=0x7ffb6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb6c009320 con 0x7ffb7c0f5480 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c106ed0 con 0x7ffb7c0f5480 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ffb6c013ec0 con 0x7ffb7c0f5480 2024-02-20T13:21:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb6c014d30 con 0x7ffb7c0f5480 2024-02-20T13:21:32.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.853+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7c107360 con 0x7ffb7c0f5480 2024-02-20T13:21:32.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.854+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ffb6c0263a0 con 0x7ffb7c0f5480 2024-02-20T13:21:32.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.854+0000 7ffb6affd700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ffb64031090 0x7ffb64033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:32.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.854+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ffb6c042440 con 0x7ffb7c0f5480 2024-02-20T13:21:32.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.854+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7c045bb0 con 0x7ffb7c0f5480 2024-02-20T13:21:32.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.854+0000 7ffb79d9b700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ffb64031090 0x7ffb64033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:32.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.855+0000 7ffb79d9b700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ffb64031090 0x7ffb64033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ffb700096a0 tx=0x7ffb70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:32.859+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ffb6c0266d0 con 0x7ffb7c0f5480 2024-02-20T13:21:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.065+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb7c107fe0 con 0x7ffb7c0f5480 2024-02-20T13:21:33.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.066+0000 7ffb6affd700 1 -- 172.21.15.114:0/567419344 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb6c02b020 con 0x7ffb7c0f5480 2024-02-20T13:21:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:33.066 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:33.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ffb64031090 msgr2=0x7ffb64033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:33.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ffb64031090 0x7ffb64033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ffb700096a0 tx=0x7ffb70006b40).stop 2024-02-20T13:21:33.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 msgr2=0x7ffb7c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c1067e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7ffb6c0089f0 tx=0x7ffb6c00b040).stop 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 shutdown_connections 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ffb64031090 0x7ffb64033550 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 --2- 172.21.15.114:0/567419344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb7c0f5480 0x7ffb7c1067e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 >> 172.21.15.114:0/567419344 conn(0x7ffb7c0f0be0 msgr2=0x7ffb7c05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 shutdown_connections 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:33.067+0000 7ffb808b1700 1 -- 172.21.15.114:0/567419344 wait complete. 2024-02-20T13:21:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:33.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:32.172516+0000 mgr.smithi038.powepi (mgr.14156) 124 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:33.066243+0000 mon.smithi038 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.114:0/567419344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:34.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:34.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:35.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:34.172844+0000 mgr.smithi038.powepi (mgr.14156) 125 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 -- 172.21.15.114:0/978376761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 msgr2=0x7f3fe40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 --2- 172.21.15.114:0/978376761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe40f58a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3fd8009fb0 tx=0x7f3fd8009b30).stop 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 -- 172.21.15.114:0/978376761 shutdown_connections 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 --2- 172.21.15.114:0/978376761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe40f58a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 -- 172.21.15.114:0/978376761 >> 172.21.15.114:0/978376761 conn(0x7f3fe40f0be0 msgr2=0x7f3fe40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 -- 172.21.15.114:0/978376761 shutdown_connections 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.640+0000 7f3fe9807700 1 -- 172.21.15.114:0/978376761 wait complete. 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe9807700 1 Processor -- start 2024-02-20T13:21:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe9807700 1 -- start start 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe9807700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe9807700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4106cd0 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47150/0 (socket says 172.21.15.114:47150) 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.641+0000 7f3fe2ffd700 1 -- 172.21.15.114:0/311521521 learned_addr learned my addr 172.21.15.114:0/311521521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fe2ffd700 1 -- 172.21.15.114:0/311521521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8005040 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fe2ffd700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe41067e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3fd80089f0 tx=0x7f3fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd8009320 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4106ed0 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fd8013ec0 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd8014d30 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.642+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe4107360 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.643+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3fd80095f0 con 0x7f3fe40f5480 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.643+0000 7f3fcb7fe700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fcc031090 0x7f3fcc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:36.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.643+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4045bb0 con 0x7f3fe40f5480 2024-02-20T13:21:36.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.643+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fd80424a0 con 0x7f3fe40f5480 2024-02-20T13:21:36.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.643+0000 7f3fe27fc700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fcc031090 0x7f3fcc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:36.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.644+0000 7f3fe27fc700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fcc031090 0x7f3fcc033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3fd40096a0 tx=0x7f3fd4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:36.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.648+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fd801d7f0 con 0x7f3fe40f5480 2024-02-20T13:21:36.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.854+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fe40faf70 con 0x7f3fe40f5480 2024-02-20T13:21:36.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.855+0000 7f3fcb7fe700 1 -- 172.21.15.114:0/311521521 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fd801d7f0 con 0x7f3fe40f5480 2024-02-20T13:21:36.857 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:36.857 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:36.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fcc031090 msgr2=0x7f3fcc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:36.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fcc031090 0x7f3fcc033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3fd40096a0 tx=0x7f3fd4006b40).stop 2024-02-20T13:21:36.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 msgr2=0x7f3fe41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe41067e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3fd80089f0 tx=0x7f3fd800b040).stop 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 shutdown_connections 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fcc031090 0x7f3fcc033550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 --2- 172.21.15.114:0/311521521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe40f5480 0x7f3fe41067e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.856+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 >> 172.21.15.114:0/311521521 conn(0x7f3fe40f0be0 msgr2=0x7f3fe405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.857+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 shutdown_connections 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:36.857+0000 7f3fe9807700 1 -- 172.21.15.114:0/311521521 wait complete. 2024-02-20T13:21:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:37.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:36.173198+0000 mgr.smithi038.powepi (mgr.14156) 126 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:36.856559+0000 mon.smithi038 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.114:0/311521521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:38.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:38.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:39.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:38.173526+0000 mgr.smithi038.powepi (mgr.14156) 127 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3633620478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 msgr2=0x7f4cb40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 --2- 172.21.15.114:0/3633620478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb40f69b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f4ca4004760 tx=0x7f4ca4009ad0).stop 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3633620478 shutdown_connections 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 --2- 172.21.15.114:0/3633620478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb40f69b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3633620478 >> 172.21.15.114:0/3633620478 conn(0x7f4cb40f1cd0 msgr2=0x7f4cb40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3633620478 shutdown_connections 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.507+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3633620478 wait complete. 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cbb8d5700 1 Processor -- start 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cbb8d5700 1 -- start start 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cbb8d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb4108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cbb8d5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4108f00 con 0x7f4cb40f6590 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cb9671700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb4108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cb9671700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb4108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47156/0 (socket says 172.21.15.114:47156) 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.508+0000 7f4cb9671700 1 -- 172.21.15.114:0/3590039431 learned_addr learned my addr 172.21.15.114:0/3590039431 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4cb9671700 1 -- 172.21.15.114:0/3590039431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4009d70 con 0x7f4cb40f6590 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4cb9671700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb4108a10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f4ca4004170 tx=0x7f4ca4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ca400e040 con 0x7f4cb40f6590 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4109100 con 0x7f4cb40f6590 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4ca4008ef0 con 0x7f4cb40f6590 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ca4014c90 con 0x7f4cb40f6590 2024-02-20T13:21:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.509+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb4109590 con 0x7f4cb40f6590 2024-02-20T13:21:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.510+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4ca4014420 con 0x7f4cb40f6590 2024-02-20T13:21:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.510+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb4045bb0 con 0x7f4cb40f6590 2024-02-20T13:21:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.510+0000 7f4ca9ffb700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4ca0031090 0x7f4ca0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.510+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4ca4042260 con 0x7f4cb40f6590 2024-02-20T13:21:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.510+0000 7f4cb8e70700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4ca0031090 0x7f4ca0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.511+0000 7f4cb8e70700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4ca0031090 0x7f4ca0033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f4cb00096a0 tx=0x7f4cb0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.515+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4ca4026300 con 0x7f4cb40f6590 2024-02-20T13:21:40.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.722+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cb410a390 con 0x7f4cb40f6590 2024-02-20T13:21:40.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.723+0000 7f4ca9ffb700 1 -- 172.21.15.114:0/3590039431 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ca4012b20 con 0x7f4cb40f6590 2024-02-20T13:21:40.725 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:40.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4ca0031090 msgr2=0x7f4ca0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:40.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4ca0031090 0x7f4ca0033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f4cb00096a0 tx=0x7f4cb0006b40).stop 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 msgr2=0x7f4cb4108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb4108a10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f4ca4004170 tx=0x7f4ca4009ad0).stop 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 shutdown_connections 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4ca0031090 0x7f4ca0033550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 --2- 172.21.15.114:0/3590039431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb40f6590 0x7f4cb4108a10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.724+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 >> 172.21.15.114:0/3590039431 conn(0x7f4cb40f1cd0 msgr2=0x7f4cb40f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.725+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 shutdown_connections 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:40.725+0000 7f4cbb8d5700 1 -- 172.21.15.114:0/3590039431 wait complete. 2024-02-20T13:21:40.727 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:41.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:40.173879+0000 mgr.smithi038.powepi (mgr.14156) 128 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:40.724551+0000 mon.smithi038 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.114:0/3590039431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:42.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:42.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:43.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:42.174146+0000 mgr.smithi038.powepi (mgr.14156) 129 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 -- 172.21.15.114:0/2136897134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 msgr2=0x7fe1ec0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 --2- 172.21.15.114:0/2136897134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec0f58a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe1e0009fb0 tx=0x7fe1e0009b30).stop 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 -- 172.21.15.114:0/2136897134 shutdown_connections 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 --2- 172.21.15.114:0/2136897134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec0f58a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 -- 172.21.15.114:0/2136897134 >> 172.21.15.114:0/2136897134 conn(0x7fe1ec0f0be0 msgr2=0x7fe1ec0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 -- 172.21.15.114:0/2136897134 shutdown_connections 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.410+0000 7fe1f0e29700 1 -- 172.21.15.114:0/2136897134 wait complete. 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1f0e29700 1 Processor -- start 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1f0e29700 1 -- start start 2024-02-20T13:21:44.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1f0e29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1f0e29700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec060320 con 0x7fe1ec0f5480 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1ea59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1ea59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47178/0 (socket says 172.21.15.114:47178) 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.411+0000 7fe1ea59c700 1 -- 172.21.15.114:0/3542548106 learned_addr learned my addr 172.21.15.114:0/3542548106 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1ea59c700 1 -- 172.21.15.114:0/3542548106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0005040 con 0x7fe1ec0f5480 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1ea59c700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec1094f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fe1e00089f0 tx=0x7fe1e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1e0009320 con 0x7fe1ec0f5480 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec060520 con 0x7fe1ec0f5480 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe1e0013ec0 con 0x7fe1ec0f5480 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1e0014d30 con 0x7fe1ec0f5480 2024-02-20T13:21:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.412+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec109ba0 con 0x7fe1ec0f5480 2024-02-20T13:21:44.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.413+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe1e00095f0 con 0x7fe1ec0f5480 2024-02-20T13:21:44.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.413+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec045bb0 con 0x7fe1ec0f5480 2024-02-20T13:21:44.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.413+0000 7fe1daffd700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1d4031090 0x7fe1d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:44.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.413+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe1e00424e0 con 0x7fe1ec0f5480 2024-02-20T13:21:44.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.413+0000 7fe1e9d9b700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1d4031090 0x7fe1d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:44.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.414+0000 7fe1e9d9b700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1d4031090 0x7fe1d4033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe1dc0096a0 tx=0x7fe1dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.418+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe1e001d7f0 con 0x7fe1ec0f5480 2024-02-20T13:21:44.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.625+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1ec0faf70 con 0x7fe1ec0f5480 2024-02-20T13:21:44.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.626+0000 7fe1daffd700 1 -- 172.21.15.114:0/3542548106 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe1e000fe40 con 0x7fe1ec0f5480 2024-02-20T13:21:44.627 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:44.628 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1d4031090 msgr2=0x7fe1d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1d4031090 0x7fe1d4033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe1dc0096a0 tx=0x7fe1dc006b40).stop 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 msgr2=0x7fe1ec1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec1094f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fe1e00089f0 tx=0x7fe1e000b040).stop 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 shutdown_connections 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1d4031090 0x7fe1d4033550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.627+0000 7fe1f0e29700 1 --2- 172.21.15.114:0/3542548106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec0f5480 0x7fe1ec1094f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.628+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 >> 172.21.15.114:0/3542548106 conn(0x7fe1ec0f0be0 msgr2=0x7fe1ec0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.628+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 shutdown_connections 2024-02-20T13:21:44.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:44.628+0000 7fe1f0e29700 1 -- 172.21.15.114:0/3542548106 wait complete. 2024-02-20T13:21:44.631 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:45.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:44.174392+0000 mgr.smithi038.powepi (mgr.14156) 130 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:45.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:44.627575+0000 mon.smithi038 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.114:0/3542548106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:46.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:46.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:47.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:46.174716+0000 mgr.smithi038.powepi (mgr.14156) 131 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:48.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.301+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/1712323393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 msgr2=0x7f0ccc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:48.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.301+0000 7f0cd2c5a700 1 --2- 172.21.15.114:0/1712323393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc0f58a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f0cc0009fb0 tx=0x7f0cc0009b30).stop 2024-02-20T13:21:48.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/1712323393 shutdown_connections 2024-02-20T13:21:48.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 --2- 172.21.15.114:0/1712323393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc0f58a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/1712323393 >> 172.21.15.114:0/1712323393 conn(0x7f0ccc0f0be0 msgr2=0x7f0ccc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/1712323393 shutdown_connections 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/1712323393 wait complete. 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 Processor -- start 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 -- start start 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.302+0000 7f0cd2c5a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc105be0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd09f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd09f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47036/0 (socket says 172.21.15.114:47036) 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd09f6700 1 -- 172.21.15.114:0/3775703521 learned_addr learned my addr 172.21.15.114:0/3775703521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd09f6700 1 -- 172.21.15.114:0/3775703521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc0005040 con 0x7f0ccc0f5480 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd09f6700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc1056f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0cc00089f0 tx=0x7f0cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc0009320 con 0x7f0ccc0f5480 2024-02-20T13:21:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc105de0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0cc0013ec0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc0014d30 con 0x7f0ccc0f5480 2024-02-20T13:21:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.303+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc106270 con 0x7f0ccc0f5480 2024-02-20T13:21:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.304+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0cc0014420 con 0x7f0ccc0f5480 2024-02-20T13:21:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.304+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc045bb0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.304+0000 7f0cc97fa700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cbc0310e0 0x7f0cbc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.304+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0cc00424a0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.305+0000 7f0ccbfff700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cbc0310e0 0x7f0cbc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.305+0000 7f0ccbfff700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cbc0310e0 0x7f0cbc0335a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f0cb80096a0 tx=0x7f0cb8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:48.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.309+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0cc001d7f0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.515+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ccc0faf70 con 0x7f0ccc0f5480 2024-02-20T13:21:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.516+0000 7f0cc97fa700 1 -- 172.21.15.114:0/3775703521 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cc001d7f0 con 0x7f0ccc0f5480 2024-02-20T13:21:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:48.518 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.517+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cbc0310e0 msgr2=0x7f0cbc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.517+0000 7f0cd2c5a700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cbc0310e0 0x7f0cbc0335a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f0cb80096a0 tx=0x7f0cb8006b40).stop 2024-02-20T13:21:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.517+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 msgr2=0x7f0ccc1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.517+0000 7f0cd2c5a700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc1056f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0cc00089f0 tx=0x7f0cc000b040).stop 2024-02-20T13:21:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.518+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 shutdown_connections 2024-02-20T13:21:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.518+0000 7f0cd2c5a700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cbc0310e0 0x7f0cbc0335a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.518+0000 7f0cd2c5a700 1 --2- 172.21.15.114:0/3775703521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ccc0f5480 0x7f0ccc1056f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.518+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 >> 172.21.15.114:0/3775703521 conn(0x7f0ccc0f0be0 msgr2=0x7f0ccc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.518+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 shutdown_connections 2024-02-20T13:21:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:48.518+0000 7f0cd2c5a700 1 -- 172.21.15.114:0/3775703521 wait complete. 2024-02-20T13:21:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:49.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:48.175066+0000 mgr.smithi038.powepi (mgr.14156) 132 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:49.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:48.517662+0000 mon.smithi038 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.114:0/3775703521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:50.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:50.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:51.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:50.175407+0000 mgr.smithi038.powepi (mgr.14156) 133 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:52.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 -- 172.21.15.114:0/1217209588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 msgr2=0x7f7e000f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:52.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 --2- 172.21.15.114:0/1217209588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e000f6800 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7dfc009fb0 tx=0x7f7dfc009b30).stop 2024-02-20T13:21:52.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 -- 172.21.15.114:0/1217209588 shutdown_connections 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 --2- 172.21.15.114:0/1217209588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e000f6800 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 -- 172.21.15.114:0/1217209588 >> 172.21.15.114:0/1217209588 conn(0x7f7e000f1b40 msgr2=0x7f7e000f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 -- 172.21.15.114:0/1217209588 shutdown_connections 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.112+0000 7f7e074e9700 1 -- 172.21.15.114:0/1217209588 wait complete. 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e074e9700 1 Processor -- start 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e074e9700 1 -- start start 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e074e9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e0006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e074e9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00069f00 con 0x7f7e000f63e0 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e064e7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e0006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e064e7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e0006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47056/0 (socket says 172.21.15.114:47056) 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e064e7700 1 -- 172.21.15.114:0/3501228128 learned_addr learned my addr 172.21.15.114:0/3501228128 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e064e7700 1 -- 172.21.15.114:0/3501228128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc005040 con 0x7f7e000f63e0 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.113+0000 7f7e064e7700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e0006ba80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7dfc0089f0 tx=0x7f7dfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.114+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dfc009320 con 0x7f7e000f63e0 2024-02-20T13:21:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.114+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0006a0a0 con 0x7f7e000f63e0 2024-02-20T13:21:52.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.114+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7dfc013ec0 con 0x7f7e000f63e0 2024-02-20T13:21:52.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.114+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dfc014d30 con 0x7f7e000f63e0 2024-02-20T13:21:52.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.114+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0006a530 con 0x7f7e000f63e0 2024-02-20T13:21:52.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.114+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7dfc0095f0 con 0x7f7e000f63e0 2024-02-20T13:21:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.115+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e000591e0 con 0x7f7e000f63e0 2024-02-20T13:21:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.115+0000 7f7df6ffd700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7dec031130 0x7f7dec0335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.115+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7dfc042450 con 0x7f7e000f63e0 2024-02-20T13:21:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.115+0000 7f7e05ce6700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7dec031130 0x7f7dec0335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.115+0000 7f7e05ce6700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7dec031130 0x7f7dec0335f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7df00096a0 tx=0x7f7df0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.119+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7dfc01dd70 con 0x7f7e000f63e0 2024-02-20T13:21:52.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.325+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e000fbed0 con 0x7f7e000f63e0 2024-02-20T13:21:52.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.326+0000 7f7df6ffd700 1 -- 172.21.15.114:0/3501228128 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7dfc01d380 con 0x7f7e000f63e0 2024-02-20T13:21:52.328 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:52.328 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:52.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.327+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7dec031130 msgr2=0x7f7dec0335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:52.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7dec031130 0x7f7dec0335f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7df00096a0 tx=0x7f7df0006b40).stop 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 msgr2=0x7f7e0006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e0006ba80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7dfc0089f0 tx=0x7f7dfc00b040).stop 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 shutdown_connections 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7dec031130 0x7f7dec0335f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 --2- 172.21.15.114:0/3501228128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e000f63e0 0x7f7e0006ba80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 >> 172.21.15.114:0/3501228128 conn(0x7f7e000f1b40 msgr2=0x7f7e000f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 shutdown_connections 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:52.328+0000 7f7e074e9700 1 -- 172.21.15.114:0/3501228128 wait complete. 2024-02-20T13:21:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:53.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:52.175704+0000 mgr.smithi038.powepi (mgr.14156) 134 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:53.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:52.327646+0000 mon.smithi038 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.114:0/3501228128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:53.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:53.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:55.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:54.176033+0000 mgr.smithi038.powepi (mgr.14156) 135 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 -- 172.21.15.114:0/2476049512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 msgr2=0x7fa2340f4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 --2- 172.21.15.114:0/2476049512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2340f4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa228009fb0 tx=0x7fa228009b30).stop 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 -- 172.21.15.114:0/2476049512 shutdown_connections 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 --2- 172.21.15.114:0/2476049512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2340f4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 -- 172.21.15.114:0/2476049512 >> 172.21.15.114:0/2476049512 conn(0x7fa2340f0390 msgr2=0x7fa2340f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 -- 172.21.15.114:0/2476049512 shutdown_connections 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.903+0000 7fa23a37d700 1 -- 172.21.15.114:0/2476049512 wait complete. 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa23a37d700 1 Processor -- start 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa23a37d700 1 -- start start 2024-02-20T13:21:55.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa23a37d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2341067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa23a37d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234106c90 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa233fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2341067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa233fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2341067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:39454/0 (socket says 172.21.15.114:39454) 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.904+0000 7fa233fff700 1 -- 172.21.15.114:0/1663020247 learned_addr learned my addr 172.21.15.114:0/1663020247 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa233fff700 1 -- 172.21.15.114:0/1663020247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228005040 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa233fff700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2341067a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fa2280089f0 tx=0x7fa22800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa228009320 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa234106e90 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa228013ec0 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa228014d30 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.905+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa234107320 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.906+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa2280095f0 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.906+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234045bb0 con 0x7fa2340f4930 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.906+0000 7fa230ff9700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa21c031090 0x7fa21c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:55.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.906+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2280423e0 con 0x7fa2340f4930 2024-02-20T13:21:55.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.906+0000 7fa2337fe700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa21c031090 0x7fa21c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:55.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.907+0000 7fa2337fe700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa21c031090 0x7fa21c033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa2240096a0 tx=0x7fa224006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:55.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:55.911+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa22801d7f0 con 0x7fa2340f4930 2024-02-20T13:21:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.118+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2341081d0 con 0x7fa2340f4930 2024-02-20T13:21:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa230ff9700 1 -- 172.21.15.114:0/1663020247 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa22801d7f0 con 0x7fa2340f4930 2024-02-20T13:21:56.120 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:56.120 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa21c031090 msgr2=0x7fa21c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa21c031090 0x7fa21c033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa2240096a0 tx=0x7fa224006b40).stop 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 msgr2=0x7fa2341067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2341067a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fa2280089f0 tx=0x7fa22800b040).stop 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 shutdown_connections 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa21c031090 0x7fa21c033550 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 --2- 172.21.15.114:0/1663020247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2340f4930 0x7fa2341067a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 >> 172.21.15.114:0/1663020247 conn(0x7fa2340f0390 msgr2=0x7fa2340f0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 shutdown_connections 2024-02-20T13:21:56.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:56.120+0000 7fa23a37d700 1 -- 172.21.15.114:0/1663020247 wait complete. 2024-02-20T13:21:56.123 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:21:56.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:56.120453+0000 mon.smithi038 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.114:0/1663020247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:21:57.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:56.176447+0000 mgr.smithi038.powepi (mgr.14156) 136 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:57.734 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:21:57.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:21:59.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 -- 172.21.15.114:0/675232348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 msgr2=0x7fbc100f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:59.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 --2- 172.21.15.114:0/675232348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc100f9160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fbc04009fb0 tx=0x7fbc04009b30).stop 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 -- 172.21.15.114:0/675232348 shutdown_connections 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 --2- 172.21.15.114:0/675232348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc100f9160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 -- 172.21.15.114:0/675232348 >> 172.21.15.114:0/675232348 conn(0x7fbc100f0be0 msgr2=0x7fbc100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 -- 172.21.15.114:0/675232348 shutdown_connections 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.587+0000 7fbc14cd0700 1 -- 172.21.15.114:0/675232348 wait complete. 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc14cd0700 1 Processor -- start 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc14cd0700 1 -- start start 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc14cd0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc10106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc14cd0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc10106c40 con 0x7fbc10060360 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc0e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc10106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc0e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc10106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:39460/0 (socket says 172.21.15.114:39460) 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc0e59c700 1 -- 172.21.15.114:0/2469124223 learned_addr learned my addr 172.21.15.114:0/2469124223 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc0e59c700 1 -- 172.21.15.114:0/2469124223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04005040 con 0x7fbc10060360 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.588+0000 7fbc0e59c700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc10106750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fbc04009f80 tx=0x7fbc0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:59.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.589+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc04009320 con 0x7fbc10060360 2024-02-20T13:21:59.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.589+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbc04013680 con 0x7fbc10060360 2024-02-20T13:21:59.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.589+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc10106e40 con 0x7fbc10060360 2024-02-20T13:21:59.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.589+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0401eda0 con 0x7fbc10060360 2024-02-20T13:21:59.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.589+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc101072d0 con 0x7fbc10060360 2024-02-20T13:21:59.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.589+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbc0401e420 con 0x7fbc10060360 2024-02-20T13:21:59.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.590+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc10045bb0 con 0x7fbc10060360 2024-02-20T13:21:59.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.590+0000 7fbbfeffd700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbf80310e0 0x7fbbf80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:21:59.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.590+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbc040423e0 con 0x7fbc10060360 2024-02-20T13:21:59.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.590+0000 7fbc0dd9b700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbf80310e0 0x7fbbf80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:21:59.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.590+0000 7fbc0dd9b700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbf80310e0 0x7fbbf80335a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fbc000096a0 tx=0x7fbc00006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:21:59.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.595+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbc0401e6b0 con 0x7fbc10060360 2024-02-20T13:21:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:21:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:21:58.176787+0000 mgr.smithi038.powepi (mgr.14156) 137 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:21:59.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.803+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc10108180 con 0x7fbc10060360 2024-02-20T13:21:59.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.804+0000 7fbbfeffd700 1 -- 172.21.15.114:0/2469124223 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc04004490 con 0x7fbc10060360 2024-02-20T13:21:59.805 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:21:59.805 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:21:59.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbf80310e0 msgr2=0x7fbbf80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:59.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbf80310e0 0x7fbbf80335a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fbc000096a0 tx=0x7fbc00006b40).stop 2024-02-20T13:21:59.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 msgr2=0x7fbc10106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:21:59.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc10106750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fbc04009f80 tx=0x7fbc0400b040).stop 2024-02-20T13:21:59.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 shutdown_connections 2024-02-20T13:21:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbf80310e0 0x7fbbf80335a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 --2- 172.21.15.114:0/2469124223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc10060360 0x7fbc10106750 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:21:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 >> 172.21.15.114:0/2469124223 conn(0x7fbc100f0be0 msgr2=0x7fbc100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:21:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 shutdown_connections 2024-02-20T13:21:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:21:59.805+0000 7fbc14cd0700 1 -- 172.21.15.114:0/2469124223 wait complete. 2024-02-20T13:21:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:00.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:21:59.804323+0000 mon.smithi038 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.114:0/2469124223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:01.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:01.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:00.177163+0000 mgr.smithi038.powepi (mgr.14156) 138 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:03.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:02.177509+0000 mgr.smithi038.powepi (mgr.14156) 139 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 -- 172.21.15.114:0/686566445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 msgr2=0x7fb3b40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 --2- 172.21.15.114:0/686566445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b40f58a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb3a8009fb0 tx=0x7fb3a8009b30).stop 2024-02-20T13:22:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 -- 172.21.15.114:0/686566445 shutdown_connections 2024-02-20T13:22:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 --2- 172.21.15.114:0/686566445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b40f58a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 -- 172.21.15.114:0/686566445 >> 172.21.15.114:0/686566445 conn(0x7fb3b40f0be0 msgr2=0x7fb3b40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 -- 172.21.15.114:0/686566445 shutdown_connections 2024-02-20T13:22:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.470+0000 7fb3b9a26700 1 -- 172.21.15.114:0/686566445 wait complete. 2024-02-20T13:22:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b9a26700 1 Processor -- start 2024-02-20T13:22:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b9a26700 1 -- start start 2024-02-20T13:22:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b9a26700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b9a26700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4060320 con 0x7fb3b40f5480 2024-02-20T13:22:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:39484/0 (socket says 172.21.15.114:39484) 2024-02-20T13:22:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b2ffd700 1 -- 172.21.15.114:0/652940365 learned_addr learned my addr 172.21.15.114:0/652940365 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b2ffd700 1 -- 172.21.15.114:0/652940365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8005040 con 0x7fb3b40f5480 2024-02-20T13:22:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.471+0000 7fb3b2ffd700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b41094f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fb3a80089f0 tx=0x7fb3a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.472+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a8009320 con 0x7fb3b40f5480 2024-02-20T13:22:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.472+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b4060520 con 0x7fb3b40f5480 2024-02-20T13:22:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.472+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb3a8013ec0 con 0x7fb3b40f5480 2024-02-20T13:22:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.472+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a8014d30 con 0x7fb3b40f5480 2024-02-20T13:22:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.472+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b4109ba0 con 0x7fb3b40f5480 2024-02-20T13:22:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.473+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb3a80263a0 con 0x7fb3b40f5480 2024-02-20T13:22:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.473+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b4045bb0 con 0x7fb3b40f5480 2024-02-20T13:22:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.473+0000 7fb39b7fe700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb39c031090 0x7fb39c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.473+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb3a8042390 con 0x7fb3b40f5480 2024-02-20T13:22:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.473+0000 7fb3b27fc700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb39c031090 0x7fb39c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.474+0000 7fb3b27fc700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb39c031090 0x7fb39c033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb3a40096a0 tx=0x7fb3a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.478+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb3a8012920 con 0x7fb3b40f5480 2024-02-20T13:22:03.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.684+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3b40faf70 con 0x7fb3b40f5480 2024-02-20T13:22:03.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.685+0000 7fb39b7fe700 1 -- 172.21.15.114:0/652940365 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3a8012330 con 0x7fb3b40f5480 2024-02-20T13:22:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:03.686 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb39c031090 msgr2=0x7fb39c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb39c031090 0x7fb39c033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb3a40096a0 tx=0x7fb3a4006b40).stop 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 msgr2=0x7fb3b41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b41094f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fb3a80089f0 tx=0x7fb3a800b040).stop 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 shutdown_connections 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb39c031090 0x7fb39c033550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 --2- 172.21.15.114:0/652940365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3b40f5480 0x7fb3b41094f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 >> 172.21.15.114:0/652940365 conn(0x7fb3b40f0be0 msgr2=0x7fb3b40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:03.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 shutdown_connections 2024-02-20T13:22:03.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:03.686+0000 7fb3b9a26700 1 -- 172.21.15.114:0/652940365 wait complete. 2024-02-20T13:22:03.688 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:04.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:03.685306+0000 mon.smithi038 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.114:0/652940365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:05.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:05.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:05.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:04.177781+0000 mgr.smithi038.powepi (mgr.14156) 140 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 -- 172.21.15.114:0/2991290403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 msgr2=0x7f41180f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 --2- 172.21.15.114:0/2991290403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41180f69b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4108004760 tx=0x7f4108009ad0).stop 2024-02-20T13:22:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 -- 172.21.15.114:0/2991290403 shutdown_connections 2024-02-20T13:22:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 --2- 172.21.15.114:0/2991290403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41180f69b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 -- 172.21.15.114:0/2991290403 >> 172.21.15.114:0/2991290403 conn(0x7f41180f1cd0 msgr2=0x7f41180f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 -- 172.21.15.114:0/2991290403 shutdown_connections 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.099+0000 7f4120777700 1 -- 172.21.15.114:0/2991290403 wait complete. 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.100+0000 7f4120777700 1 Processor -- start 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.100+0000 7f4120777700 1 -- start start 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.100+0000 7f4120777700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41181067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.100+0000 7f4120777700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118106cb0 con 0x7f41180f6590 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f411e513700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41181067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f411e513700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41181067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37186/0 (socket says 172.21.15.114:37186) 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f411e513700 1 -- 172.21.15.114:0/3014306738 learned_addr learned my addr 172.21.15.114:0/3014306738 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f411e513700 1 -- 172.21.15.114:0/3014306738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108009d70 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f411e513700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41181067c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4108004170 tx=0x7f4108009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f410800e040 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4118106eb0 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4108008ef0 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4108014c90 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.101+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118107340 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.102+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4108014420 con 0x7f41180f6590 2024-02-20T13:22:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.102+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4118045bb0 con 0x7f41180f6590 2024-02-20T13:22:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.102+0000 7f410effd700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4104031090 0x7f4104033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.102+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f41080422b0 con 0x7f41180f6590 2024-02-20T13:22:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.102+0000 7f411dd12700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4104031090 0x7f4104033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.103+0000 7f411dd12700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4104031090 0x7f4104033550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f41140096a0 tx=0x7f4114006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.107+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4108026300 con 0x7f41180f6590 2024-02-20T13:22:07.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.313+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41181081f0 con 0x7f41180f6590 2024-02-20T13:22:07.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.314+0000 7f410effd700 1 -- 172.21.15.114:0/3014306738 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4108012b20 con 0x7f41180f6590 2024-02-20T13:22:07.315 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:07.315 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4104031090 msgr2=0x7f4104033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4104031090 0x7f4104033550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f41140096a0 tx=0x7f4114006b40).stop 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 msgr2=0x7f41181067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41181067c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4108004170 tx=0x7f4108009ad0).stop 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 shutdown_connections 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4104031090 0x7f4104033550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:07.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 --2- 172.21.15.114:0/3014306738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41180f6590 0x7f41181067c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:07.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.315+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 >> 172.21.15.114:0/3014306738 conn(0x7f41180f1cd0 msgr2=0x7f41180f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:07.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.316+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 shutdown_connections 2024-02-20T13:22:07.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:07.316+0000 7f4120777700 1 -- 172.21.15.114:0/3014306738 wait complete. 2024-02-20T13:22:07.317 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:07.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:06.178104+0000 mgr.smithi038.powepi (mgr.14156) 141 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:08.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:07.314617+0000 mon.smithi038 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.114:0/3014306738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:09.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:09.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:09.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:08.178455+0000 mgr.smithi038.powepi (mgr.14156) 142 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 -- 172.21.15.114:0/1106802060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 msgr2=0x7feed40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 --2- 172.21.15.114:0/1106802060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed40f58a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7feec4009fb0 tx=0x7feec4009b30).stop 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 -- 172.21.15.114:0/1106802060 shutdown_connections 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 --2- 172.21.15.114:0/1106802060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed40f58a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 -- 172.21.15.114:0/1106802060 >> 172.21.15.114:0/1106802060 conn(0x7feed40f0be0 msgr2=0x7feed40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 -- 172.21.15.114:0/1106802060 shutdown_connections 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.032+0000 7feed936a700 1 -- 172.21.15.114:0/1106802060 wait complete. 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.033+0000 7feed936a700 1 Processor -- start 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.033+0000 7feed936a700 1 -- start start 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.033+0000 7feed936a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed41056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.033+0000 7feed936a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed4105be0 con 0x7feed40f5480 2024-02-20T13:22:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed41056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed41056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37208/0 (socket says 172.21.15.114:37208) 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed2ffd700 1 -- 172.21.15.114:0/1377000574 learned_addr learned my addr 172.21.15.114:0/1377000574 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed2ffd700 1 -- 172.21.15.114:0/1377000574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4005040 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed2ffd700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed41056f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7feec40089f0 tx=0x7feec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec4009320 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed4105de0 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7feec4013ec0 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec4014d30 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.034+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed4106270 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.035+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7feec40095f0 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.035+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed4045bb0 con 0x7feed40f5480 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.035+0000 7feebb7fe700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feebc031090 0x7feebc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.035+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7feec40423e0 con 0x7feed40f5480 2024-02-20T13:22:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.035+0000 7feed27fc700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feebc031090 0x7feebc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.036+0000 7feed27fc700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feebc031090 0x7feebc033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7feec80096a0 tx=0x7feec8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.040+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7feec401d980 con 0x7feed40f5480 2024-02-20T13:22:11.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.250+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feed40faf70 con 0x7feed40f5480 2024-02-20T13:22:11.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.251+0000 7feebb7fe700 1 -- 172.21.15.114:0/1377000574 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feec401d380 con 0x7feed40f5480 2024-02-20T13:22:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:11.253 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:11.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feebc031090 msgr2=0x7feebc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:11.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feebc031090 0x7feebc033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7feec80096a0 tx=0x7feec8006b40).stop 2024-02-20T13:22:11.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 msgr2=0x7feed41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed41056f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7feec40089f0 tx=0x7feec400b040).stop 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 shutdown_connections 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7feebc031090 0x7feebc033550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 --2- 172.21.15.114:0/1377000574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40f5480 0x7feed41056f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 >> 172.21.15.114:0/1377000574 conn(0x7feed40f0be0 msgr2=0x7feed40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 shutdown_connections 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:11.253+0000 7feed936a700 1 -- 172.21.15.114:0/1377000574 wait complete. 2024-02-20T13:22:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:11.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:10.178799+0000 mgr.smithi038.powepi (mgr.14156) 143 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:12.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:11.251685+0000 mon.smithi038 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.114:0/1377000574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:12.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:12.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:13.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:12.179186+0000 mgr.smithi038.powepi (mgr.14156) 144 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 -- 172.21.15.114:0/312698608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 msgr2=0x7f95c00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 --2- 172.21.15.114:0/312698608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c00f58a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f95b4009fb0 tx=0x7f95b4009b30).stop 2024-02-20T13:22:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 -- 172.21.15.114:0/312698608 shutdown_connections 2024-02-20T13:22:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 --2- 172.21.15.114:0/312698608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c00f58a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 -- 172.21.15.114:0/312698608 >> 172.21.15.114:0/312698608 conn(0x7f95c00f0be0 msgr2=0x7f95c00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 -- 172.21.15.114:0/312698608 shutdown_connections 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 -- 172.21.15.114:0/312698608 wait complete. 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.918+0000 7f95c65d9700 1 Processor -- start 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.919+0000 7f95c65d9700 1 -- start start 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.919+0000 7f95c65d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.919+0000 7f95c65d9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c0060300 con 0x7f95c00f5480 2024-02-20T13:22:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.919+0000 7f95bffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.919+0000 7f95bffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:37238/0 (socket says 172.21.15.114:37238) 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.919+0000 7f95bffff700 1 -- 172.21.15.114:0/2011263935 learned_addr learned my addr 172.21.15.114:0/2011263935 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95bffff700 1 -- 172.21.15.114:0/2011263935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b4005040 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95bffff700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c0109510 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f95b40089f0 tx=0x7f95b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b4009320 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c0060500 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f95b4013ec0 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b4014d30 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.920+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c0109bc0 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.921+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f95b40095f0 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.921+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c00591e0 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.921+0000 7f95bcff9700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f95a8031090 0x7f95a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.921+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f95b4042420 con 0x7f95c00f5480 2024-02-20T13:22:14.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.921+0000 7f95bf7fe700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f95a8031090 0x7f95a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:14.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.922+0000 7f95bf7fe700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f95a8031090 0x7f95a8033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f95b00096a0 tx=0x7f95b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:14.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:14.926+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f95b401d7f0 con 0x7f95c00f5480 2024-02-20T13:22:15.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.139+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95c0045bb0 con 0x7f95c00f5480 2024-02-20T13:22:15.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.140+0000 7f95bcff9700 1 -- 172.21.15.114:0/2011263935 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95b400fe40 con 0x7f95c00f5480 2024-02-20T13:22:15.141 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:15.141 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:15.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f95a8031090 msgr2=0x7f95a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:15.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f95a8031090 0x7f95a8033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f95b00096a0 tx=0x7f95b0006b40).stop 2024-02-20T13:22:15.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 msgr2=0x7f95c0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:15.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c0109510 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f95b40089f0 tx=0x7f95b400b040).stop 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 shutdown_connections 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f95a8031090 0x7f95a8033550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 --2- 172.21.15.114:0/2011263935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c00f5480 0x7f95c0109510 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 >> 172.21.15.114:0/2011263935 conn(0x7f95c00f0be0 msgr2=0x7f95c00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 shutdown_connections 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:15.142+0000 7f95c65d9700 1 -- 172.21.15.114:0/2011263935 wait complete. 2024-02-20T13:22:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:15.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:14.179504+0000 mgr.smithi038.powepi (mgr.14156) 145 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:15.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:15.140640+0000 mon.smithi038 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.114:0/2011263935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:16.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:16.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:17.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:16.179909+0000 mgr.smithi038.powepi (mgr.14156) 146 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:18.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 -- 172.21.15.114:0/3434018114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 msgr2=0x7fe1fc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:18.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 --2- 172.21.15.114:0/3434018114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc0f56f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fe1f0009fb0 tx=0x7fe1f0009b30).stop 2024-02-20T13:22:18.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 -- 172.21.15.114:0/3434018114 shutdown_connections 2024-02-20T13:22:18.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 --2- 172.21.15.114:0/3434018114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc0f56f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:18.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 -- 172.21.15.114:0/3434018114 >> 172.21.15.114:0/3434018114 conn(0x7fe1fc0f0a30 msgr2=0x7fe1fc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 -- 172.21.15.114:0/3434018114 shutdown_connections 2024-02-20T13:22:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 -- 172.21.15.114:0/3434018114 wait complete. 2024-02-20T13:22:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.812+0000 7fe201351700 1 Processor -- start 2024-02-20T13:22:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe201351700 1 -- start start 2024-02-20T13:22:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe201351700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc108800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe201351700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc108cf0 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe1fbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc108800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe1fbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc108800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55084/0 (socket says 172.21.15.114:55084) 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe1fbfff700 1 -- 172.21.15.114:0/2546345189 learned_addr learned my addr 172.21.15.114:0/2546345189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe1fbfff700 1 -- 172.21.15.114:0/2546345189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f0005040 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.813+0000 7fe1fbfff700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc108800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fe1f00089f0 tx=0x7fe1f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.814+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1f0009320 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.814+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc108ef0 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.814+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe1f0013ec0 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.814+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1f0014d30 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.814+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc109380 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.815+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe1f00095f0 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.815+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc045bb0 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.815+0000 7fe1f8ff9700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1e4031090 0x7fe1e4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:18.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.815+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe1f0042420 con 0x7fe1fc0f52d0 2024-02-20T13:22:18.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.815+0000 7fe1fb7fe700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1e4031090 0x7fe1e4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:18.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.815+0000 7fe1fb7fe700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1e4031090 0x7fe1e4033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe1ec0096a0 tx=0x7fe1ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:18.819+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe1f001d7f0 con 0x7fe1fc0f52d0 2024-02-20T13:22:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.024+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1fc10a1d0 con 0x7fe1fc0f52d0 2024-02-20T13:22:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.025+0000 7fe1f8ff9700 1 -- 172.21.15.114:0/2546345189 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe1f000fe40 con 0x7fe1fc0f52d0 2024-02-20T13:22:19.026 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:19.026 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1e4031090 msgr2=0x7fe1e4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1e4031090 0x7fe1e4033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe1ec0096a0 tx=0x7fe1ec006b40).stop 2024-02-20T13:22:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 msgr2=0x7fe1fc108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc108800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fe1f00089f0 tx=0x7fe1f000b040).stop 2024-02-20T13:22:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 shutdown_connections 2024-02-20T13:22:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe1e4031090 0x7fe1e4033550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 --2- 172.21.15.114:0/2546345189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1fc0f52d0 0x7fe1fc108800 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.026+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 >> 172.21.15.114:0/2546345189 conn(0x7fe1fc0f0a30 msgr2=0x7fe1fc05fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.027+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 shutdown_connections 2024-02-20T13:22:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:19.027+0000 7fe201351700 1 -- 172.21.15.114:0/2546345189 wait complete. 2024-02-20T13:22:19.028 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:19.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:18.180245+0000 mgr.smithi038.powepi (mgr.14156) 147 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:19.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:19.025604+0000 mon.smithi038 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.114:0/2546345189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:20.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:20.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:21.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:20.180589+0000 mgr.smithi038.powepi (mgr.14156) 148 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:22.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 -- 172.21.15.114:0/1171137356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 msgr2=0x7f94c00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:22.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 --2- 172.21.15.114:0/1171137356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c00f58a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f94b4009fb0 tx=0x7f94b4009b30).stop 2024-02-20T13:22:22.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 -- 172.21.15.114:0/1171137356 shutdown_connections 2024-02-20T13:22:22.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 --2- 172.21.15.114:0/1171137356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c00f58a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:22.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 -- 172.21.15.114:0/1171137356 >> 172.21.15.114:0/1171137356 conn(0x7f94c00f0be0 msgr2=0x7f94c00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:22.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 -- 172.21.15.114:0/1171137356 shutdown_connections 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 -- 172.21.15.114:0/1171137356 wait complete. 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.670+0000 7f94c55dc700 1 Processor -- start 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94c55dc700 1 -- start start 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94c55dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94c55dc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0106cd0 con 0x7f94c00f5480 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94beffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94beffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55092/0 (socket says 172.21.15.114:55092) 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94beffd700 1 -- 172.21.15.114:0/2850936636 learned_addr learned my addr 172.21.15.114:0/2850936636 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94beffd700 1 -- 172.21.15.114:0/2850936636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b4005040 con 0x7f94c00f5480 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94beffd700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c01067e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f94b40089f0 tx=0x7f94b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94b4009320 con 0x7f94c00f5480 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c0106ed0 con 0x7f94c00f5480 2024-02-20T13:22:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f94b4013ec0 con 0x7f94c00f5480 2024-02-20T13:22:22.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.671+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94b4014d30 con 0x7f94c00f5480 2024-02-20T13:22:22.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.672+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c0107360 con 0x7f94c00f5480 2024-02-20T13:22:22.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.672+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f94b4014420 con 0x7f94c00f5480 2024-02-20T13:22:22.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.672+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c0045bb0 con 0x7f94c00f5480 2024-02-20T13:22:22.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.672+0000 7f94a77fe700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94a80310e0 0x7f94a80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:22.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.672+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f94b4042470 con 0x7f94c00f5480 2024-02-20T13:22:22.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.673+0000 7f94be7fc700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94a80310e0 0x7f94a80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:22.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.673+0000 7f94be7fc700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94a80310e0 0x7f94a80335a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f94b00096a0 tx=0x7f94b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:22.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.677+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f94b401d980 con 0x7f94c00f5480 2024-02-20T13:22:22.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.885+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94c0108210 con 0x7f94c00f5480 2024-02-20T13:22:22.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.886+0000 7f94a77fe700 1 -- 172.21.15.114:0/2850936636 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94b401d380 con 0x7f94c00f5480 2024-02-20T13:22:22.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:22.888 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:22.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.887+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94a80310e0 msgr2=0x7f94a80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:22.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.887+0000 7f94c55dc700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94a80310e0 0x7f94a80335a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f94b00096a0 tx=0x7f94b0006b40).stop 2024-02-20T13:22:22.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.887+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 msgr2=0x7f94c01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.887+0000 7f94c55dc700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c01067e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f94b40089f0 tx=0x7f94b400b040).stop 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.888+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 shutdown_connections 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.888+0000 7f94c55dc700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94a80310e0 0x7f94a80335a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.888+0000 7f94c55dc700 1 --2- 172.21.15.114:0/2850936636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94c00f5480 0x7f94c01067e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.888+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 >> 172.21.15.114:0/2850936636 conn(0x7f94c00f0be0 msgr2=0x7f94c005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.888+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 shutdown_connections 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:22.888+0000 7f94c55dc700 1 -- 172.21.15.114:0/2850936636 wait complete. 2024-02-20T13:22:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:23.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:22.180961+0000 mgr.smithi038.powepi (mgr.14156) 149 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:23.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:22.887403+0000 mon.smithi038 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.114:0/2850936636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:24.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:24.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:24.181314+0000 mgr.smithi038.powepi (mgr.14156) 150 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 -- 172.21.15.114:0/2881933092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 msgr2=0x7fc8140fb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 --2- 172.21.15.114:0/2881933092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8140fb430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fc808009fb0 tx=0x7fc808009b30).stop 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 -- 172.21.15.114:0/2881933092 shutdown_connections 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 --2- 172.21.15.114:0/2881933092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8140fb430 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 -- 172.21.15.114:0/2881933092 >> 172.21.15.114:0/2881933092 conn(0x7fc8140f0a30 msgr2=0x7fc8140f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 -- 172.21.15.114:0/2881933092 shutdown_connections 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.379+0000 7fc818814700 1 -- 172.21.15.114:0/2881933092 wait complete. 2024-02-20T13:22:26.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc818814700 1 Processor -- start 2024-02-20T13:22:26.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc818814700 1 -- start start 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc818814700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8141087c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc818814700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc814108cb0 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc8137fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8141087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc8137fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8141087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35364/0 (socket says 172.21.15.114:35364) 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc8137fe700 1 -- 172.21.15.114:0/2606035394 learned_addr learned my addr 172.21.15.114:0/2606035394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.380+0000 7fc8137fe700 1 -- 172.21.15.114:0/2606035394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808005040 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.381+0000 7fc8137fe700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8141087c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc808009f80 tx=0x7fc80800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.381+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc808009320 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.381+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc814108eb0 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.381+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc808013680 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.381+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80801eda0 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.381+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc814109340 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.382+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc808014420 con 0x7fc8140602e0 2024-02-20T13:22:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.382+0000 7fc7fbfff700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7fc031090 0x7fc7fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:26.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.382+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc814045bb0 con 0x7fc8140602e0 2024-02-20T13:22:26.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.382+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc808042370 con 0x7fc8140602e0 2024-02-20T13:22:26.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.382+0000 7fc812ffd700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7fc031090 0x7fc7fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:26.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.382+0000 7fc812ffd700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7fc031090 0x7fc7fc033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc8040096a0 tx=0x7fc804006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:26.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.387+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc80801c910 con 0x7fc8140602e0 2024-02-20T13:22:26.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.591+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8140fe1a0 con 0x7fc8140602e0 2024-02-20T13:22:26.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.592+0000 7fc7fbfff700 1 -- 172.21.15.114:0/2606035394 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc80803b030 con 0x7fc8140602e0 2024-02-20T13:22:26.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:26.594 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:26.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7fc031090 msgr2=0x7fc7fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7fc031090 0x7fc7fc033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc8040096a0 tx=0x7fc804006b40).stop 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 msgr2=0x7fc8141087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8141087c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc808009f80 tx=0x7fc80800b040).stop 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 shutdown_connections 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7fc031090 0x7fc7fc033550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 --2- 172.21.15.114:0/2606035394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8140602e0 0x7fc8141087c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 >> 172.21.15.114:0/2606035394 conn(0x7fc8140f0a30 msgr2=0x7fc8140f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 shutdown_connections 2024-02-20T13:22:26.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:26.594+0000 7fc818814700 1 -- 172.21.15.114:0/2606035394 wait complete. 2024-02-20T13:22:26.597 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:26.181605+0000 mgr.smithi038.powepi (mgr.14156) 151 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:26.593813+0000 mon.smithi038 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.114:0/2606035394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:28.244 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:28.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:28.182023+0000 mgr.smithi038.powepi (mgr.14156) 152 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:28.278929+0000 mon.smithi038 (mon.0) 186 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:22:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:28.279175+0000 mon.smithi038 (mon.0) 187 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:22:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.136+0000 7f174c45d700 1 -- 172.21.15.114:0/2411643682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 msgr2=0x7f17440f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.136+0000 7f174c45d700 1 --2- 172.21.15.114:0/2411643682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17440f69a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1734009fb0 tx=0x7f1734009b30).stop 2024-02-20T13:22:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.136+0000 7f174c45d700 1 -- 172.21.15.114:0/2411643682 shutdown_connections 2024-02-20T13:22:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.136+0000 7f174c45d700 1 --2- 172.21.15.114:0/2411643682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17440f69a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.136+0000 7f174c45d700 1 -- 172.21.15.114:0/2411643682 >> 172.21.15.114:0/2411643682 conn(0x7f17440f1ce0 msgr2=0x7f17440f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174c45d700 1 -- 172.21.15.114:0/2411643682 shutdown_connections 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174c45d700 1 -- 172.21.15.114:0/2411643682 wait complete. 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174c45d700 1 Processor -- start 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174c45d700 1 -- start start 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174c45d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17441067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174c45d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1744106ca0 con 0x7f17440f6580 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174a1f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17441067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174a1f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17441067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35372/0 (socket says 172.21.15.114:35372) 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.137+0000 7f174a1f9700 1 -- 172.21.15.114:0/194154616 learned_addr learned my addr 172.21.15.114:0/194154616 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f174a1f9700 1 -- 172.21.15.114:0/194154616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1734005040 con 0x7f17440f6580 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f174a1f9700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17441067b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f17340089f0 tx=0x7f173400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1734009320 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1744106ea0 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1734013ec0 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1734014d30 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.138+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1744107330 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.139+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f17340095f0 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.139+0000 7f173affd700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1730031090 0x7f1730033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.139+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f17340424a0 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.139+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1744045bb0 con 0x7f17440f6580 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.139+0000 7f17499f8700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1730031090 0x7f1730033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.140+0000 7f17499f8700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1730031090 0x7f1730033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f17400096a0 tx=0x7f1740006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.144+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f173401d7f0 con 0x7f17440f6580 2024-02-20T13:22:30.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.351+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17440fc160 con 0x7f17440f6580 2024-02-20T13:22:30.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.351+0000 7f173affd700 1 -- 172.21.15.114:0/194154616 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f173401d7f0 con 0x7f17440f6580 2024-02-20T13:22:30.353 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:30.354 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1730031090 msgr2=0x7f1730033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1730031090 0x7f1730033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f17400096a0 tx=0x7f1740006b40).stop 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 msgr2=0x7f17441067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17441067b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f17340089f0 tx=0x7f173400b040).stop 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 shutdown_connections 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1730031090 0x7f1730033550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 --2- 172.21.15.114:0/194154616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17440f6580 0x7f17441067b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 >> 172.21.15.114:0/194154616 conn(0x7f17440f1ce0 msgr2=0x7f17440f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:30.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 shutdown_connections 2024-02-20T13:22:30.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:30.353+0000 7f174c45d700 1 -- 172.21.15.114:0/194154616 wait complete. 2024-02-20T13:22:30.356 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:30.182445+0000 mgr.smithi038.powepi (mgr.14156) 153 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:31.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:30.353028+0000 mon.smithi038 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.114:0/194154616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:32.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:32.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:33.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:32.182766+0000 mgr.smithi038.powepi (mgr.14156) 154 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:34.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.081+0000 7f18cc765700 1 -- 172.21.15.114:0/1886011565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 msgr2=0x7f18c40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:34.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.081+0000 7f18cc765700 1 --2- 172.21.15.114:0/1886011565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c40f58a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f18b4009fb0 tx=0x7f18b4009b30).stop 2024-02-20T13:22:34.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.081+0000 7f18cc765700 1 -- 172.21.15.114:0/1886011565 shutdown_connections 2024-02-20T13:22:34.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.081+0000 7f18cc765700 1 --2- 172.21.15.114:0/1886011565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c40f58a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:34.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.081+0000 7f18cc765700 1 -- 172.21.15.114:0/1886011565 >> 172.21.15.114:0/1886011565 conn(0x7f18c40f0be0 msgr2=0x7f18c40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:34.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.081+0000 7f18cc765700 1 -- 172.21.15.114:0/1886011565 shutdown_connections 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18cc765700 1 -- 172.21.15.114:0/1886011565 wait complete. 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18cc765700 1 Processor -- start 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18cc765700 1 -- start start 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18cc765700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c41056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18cc765700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4105be0 con 0x7f18c40f5480 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18ca501700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c41056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18ca501700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c41056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35390/0 (socket says 172.21.15.114:35390) 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.082+0000 7f18ca501700 1 -- 172.21.15.114:0/929634602 learned_addr learned my addr 172.21.15.114:0/929634602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18ca501700 1 -- 172.21.15.114:0/929634602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b4005040 con 0x7f18c40f5480 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18ca501700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c41056f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f18b40089f0 tx=0x7f18b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b4009320 con 0x7f18c40f5480 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c4105de0 con 0x7f18c40f5480 2024-02-20T13:22:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f18b4013ec0 con 0x7f18c40f5480 2024-02-20T13:22:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b4014d30 con 0x7f18c40f5480 2024-02-20T13:22:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.083+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c4106270 con 0x7f18c40f5480 2024-02-20T13:22:34.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.084+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f18b40095f0 con 0x7f18c40f5480 2024-02-20T13:22:34.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.084+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4045bb0 con 0x7f18c40f5480 2024-02-20T13:22:34.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.084+0000 7f18baffd700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f18b00310e0 0x7f18b00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:34.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.084+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f18b4042450 con 0x7f18c40f5480 2024-02-20T13:22:34.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.084+0000 7f18c9d00700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f18b00310e0 0x7f18b00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:34.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.085+0000 7f18c9d00700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f18b00310e0 0x7f18b00335a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f18c00096a0 tx=0x7f18c0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:34.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.089+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f18b40268c0 con 0x7f18c40f5480 2024-02-20T13:22:34.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.296+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18c40faf70 con 0x7f18c40f5480 2024-02-20T13:22:34.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.297+0000 7f18baffd700 1 -- 172.21.15.114:0/929634602 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18b401d380 con 0x7f18c40f5480 2024-02-20T13:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:34.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f18b00310e0 msgr2=0x7f18b00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:34.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f18b00310e0 0x7f18b00335a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f18c00096a0 tx=0x7f18c0006b40).stop 2024-02-20T13:22:34.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 msgr2=0x7f18c41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:34.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c41056f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f18b40089f0 tx=0x7f18b400b040).stop 2024-02-20T13:22:34.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 shutdown_connections 2024-02-20T13:22:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f18b00310e0 0x7f18b00335a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 --2- 172.21.15.114:0/929634602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c40f5480 0x7f18c41056f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 >> 172.21.15.114:0/929634602 conn(0x7f18c40f0be0 msgr2=0x7f18c40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 shutdown_connections 2024-02-20T13:22:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:34.299+0000 7f18cc765700 1 -- 172.21.15.114:0/929634602 wait complete. 2024-02-20T13:22:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:34.183100+0000 mgr.smithi038.powepi (mgr.14156) 155 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:34.298709+0000 mon.smithi038 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.114:0/929634602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:35.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:35.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:37.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:36.183464+0000 mgr.smithi038.powepi (mgr.14156) 156 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 -- 172.21.15.114:0/2470677124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 msgr2=0x7f5b080f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 --2- 172.21.15.114:0/2470677124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b080f69b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5af8009fb0 tx=0x7f5af8009b30).stop 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 -- 172.21.15.114:0/2470677124 shutdown_connections 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 --2- 172.21.15.114:0/2470677124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b080f69b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 -- 172.21.15.114:0/2470677124 >> 172.21.15.114:0/2470677124 conn(0x7f5b080f1cf0 msgr2=0x7f5b080f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 -- 172.21.15.114:0/2470677124 shutdown_connections 2024-02-20T13:22:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.880+0000 7f5b0e71d700 1 -- 172.21.15.114:0/2470677124 wait complete. 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.881+0000 7f5b0e71d700 1 Processor -- start 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.881+0000 7f5b0e71d700 1 -- start start 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.881+0000 7f5b0e71d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b08106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.881+0000 7f5b0e71d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08106c70 con 0x7f5b080f6590 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b07fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b08106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b07fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b08106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34122/0 (socket says 172.21.15.114:34122) 2024-02-20T13:22:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b07fff700 1 -- 172.21.15.114:0/628400412 learned_addr learned my addr 172.21.15.114:0/628400412 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b07fff700 1 -- 172.21.15.114:0/628400412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af8005040 con 0x7f5b080f6590 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b07fff700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b08106780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5af80089f0 tx=0x7f5af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5af8009320 con 0x7f5b080f6590 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b08106e70 con 0x7f5b080f6590 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5af8013ec0 con 0x7f5b080f6590 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5af8014d30 con 0x7f5b080f6590 2024-02-20T13:22:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.882+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b08107300 con 0x7f5b080f6590 2024-02-20T13:22:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.883+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5af8014420 con 0x7f5b080f6590 2024-02-20T13:22:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.883+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08045bb0 con 0x7f5b080f6590 2024-02-20T13:22:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.883+0000 7f5b04ff9700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5af4031090 0x7f5af4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.883+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5af80423e0 con 0x7f5b080f6590 2024-02-20T13:22:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.884+0000 7f5b077fe700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5af4031090 0x7f5af4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.884+0000 7f5b077fe700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5af4031090 0x7f5af4033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5af00096a0 tx=0x7f5af0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:37.888+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5af801d7f0 con 0x7f5b080f6590 2024-02-20T13:22:38.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.095+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b081081b0 con 0x7f5b080f6590 2024-02-20T13:22:38.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.095+0000 7f5b04ff9700 1 -- 172.21.15.114:0/628400412 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5af801d7f0 con 0x7f5b080f6590 2024-02-20T13:22:38.097 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:38.097 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5af4031090 msgr2=0x7f5af4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5af4031090 0x7f5af4033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5af00096a0 tx=0x7f5af0006b40).stop 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 msgr2=0x7f5b08106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b08106780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5af80089f0 tx=0x7f5af800b040).stop 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 shutdown_connections 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5af4031090 0x7f5af4033550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 --2- 172.21.15.114:0/628400412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b080f6590 0x7f5b08106780 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 >> 172.21.15.114:0/628400412 conn(0x7f5b080f1cf0 msgr2=0x7f5b080f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 shutdown_connections 2024-02-20T13:22:38.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:38.097+0000 7f5b0e71d700 1 -- 172.21.15.114:0/628400412 wait complete. 2024-02-20T13:22:38.100 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:38.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:38 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:38.097105+0000 mon.smithi038 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.114:0/628400412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:39.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:39.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:39.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:38.183798+0000 mgr.smithi038.powepi (mgr.14156) 157 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.588+0000 7f9511842700 1 -- 172.21.15.114:0/3266331653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 msgr2=0x7f950c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.588+0000 7f9511842700 1 --2- 172.21.15.114:0/3266331653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c0f69b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f94fc009fb0 tx=0x7f94fc009b30).stop 2024-02-20T13:22:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.588+0000 7f9511842700 1 -- 172.21.15.114:0/3266331653 shutdown_connections 2024-02-20T13:22:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.588+0000 7f9511842700 1 --2- 172.21.15.114:0/3266331653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c0f69b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.588+0000 7f9511842700 1 -- 172.21.15.114:0/3266331653 >> 172.21.15.114:0/3266331653 conn(0x7f950c0f1cf0 msgr2=0x7f950c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.588+0000 7f9511842700 1 -- 172.21.15.114:0/3266331653 shutdown_connections 2024-02-20T13:22:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.589+0000 7f9511842700 1 -- 172.21.15.114:0/3266331653 wait complete. 2024-02-20T13:22:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.589+0000 7f9511842700 1 Processor -- start 2024-02-20T13:22:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.589+0000 7f9511842700 1 -- start start 2024-02-20T13:22:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f9511842700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f9511842700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c106cf0 con 0x7f950c0f6590 2024-02-20T13:22:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f950affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f950affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34130/0 (socket says 172.21.15.114:34130) 2024-02-20T13:22:41.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f950affd700 1 -- 172.21.15.114:0/3279784209 learned_addr learned my addr 172.21.15.114:0/3279784209 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:41.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f950affd700 1 -- 172.21.15.114:0/3279784209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc005040 con 0x7f950c0f6590 2024-02-20T13:22:41.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.590+0000 7f950affd700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c106800 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f94fc0089f0 tx=0x7f94fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:41.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.591+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94fc009320 con 0x7f950c0f6590 2024-02-20T13:22:41.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.591+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c106ef0 con 0x7f950c0f6590 2024-02-20T13:22:41.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.591+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f94fc013ec0 con 0x7f950c0f6590 2024-02-20T13:22:41.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.591+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94fc014d30 con 0x7f950c0f6590 2024-02-20T13:22:41.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.591+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c107380 con 0x7f950c0f6590 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.592+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f94fc0263a0 con 0x7f950c0f6590 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.592+0000 7f94f37fe700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94f4031090 0x7f94f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.592+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f94fc0423a0 con 0x7f950c0f6590 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.592+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c045bb0 con 0x7f950c0f6590 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.592+0000 7f950a7fc700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94f4031090 0x7f94f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.592+0000 7f950a7fc700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94f4031090 0x7f94f4033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f95000096a0 tx=0x7f9500006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:41.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.597+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f94fc01cb10 con 0x7f950c0f6590 2024-02-20T13:22:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:40.184078+0000 mgr.smithi038.powepi (mgr.14156) 158 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:41.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.804+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f950c0fc080 con 0x7f950c0f6590 2024-02-20T13:22:41.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.805+0000 7f94f37fe700 1 -- 172.21.15.114:0/3279784209 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94fc02b020 con 0x7f950c0f6590 2024-02-20T13:22:41.806 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:41.807 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94f4031090 msgr2=0x7f94f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94f4031090 0x7f94f4033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f95000096a0 tx=0x7f9500006b40).stop 2024-02-20T13:22:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 msgr2=0x7f950c106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c106800 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f94fc0089f0 tx=0x7f94fc00b040).stop 2024-02-20T13:22:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 shutdown_connections 2024-02-20T13:22:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f94f4031090 0x7f94f4033550 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 --2- 172.21.15.114:0/3279784209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c0f6590 0x7f950c106800 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 >> 172.21.15.114:0/3279784209 conn(0x7f950c0f1cf0 msgr2=0x7f950c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 shutdown_connections 2024-02-20T13:22:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:41.806+0000 7f9511842700 1 -- 172.21.15.114:0/3279784209 wait complete. 2024-02-20T13:22:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:42.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:41.806369+0000 mon.smithi038 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.114:0/3279784209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:43.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:43.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:43.658 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:42.184442+0000 mgr.smithi038.powepi (mgr.14156) 159 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:43.659 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:42.883834+0000 mon.smithi038 (mon.0) 192 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T13:22:43.659 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:42.888301+0000 mon.smithi038 (mon.0) 193 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T13:22:43.659 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:42.889130+0000 mon.smithi038 (mon.0) 194 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:22:44.447 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:22:42.881748+0000 mgr.smithi038.powepi (mgr.14156) 160 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-alertmanager.smithi038 2024-02-20T13:22:44.447 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-alertmanager.smithi038 2024-02-20T13:22:44.447 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon alertmanager.smithi038 ... 2024-02-20T13:22:44.447 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T13:22:44.447 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:18:21Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:18:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:19:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:20:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:21:27Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:22:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:22:44.448 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('uid/gid not found') 2024-02-20T13:22:44.449 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: uid/gid not found 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: yield (conn, connr) 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: code, '\n'.join(err))) 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-alertmanager.smithi038 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-alertmanager.smithi038 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon alertmanager.smithi038 ... 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:18:21Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:18:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:19:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:22:44.450 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:20:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:21:27Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:22:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:22:44.451 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('uid/gid not found') 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: uid/gid not found 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:22:42.882025+0000 mgr.smithi038.powepi (mgr.14156) 161 : cephadm [ERR] Failed while placing alertmanager.smithi038 on smithi038: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-alertmanager.smithi038 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-alertmanager.smithi038 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon alertmanager.smithi038 ... 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:18:21Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.452 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:18:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:19:36Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:20:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:21:27Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:22:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:22:44.453 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('uid/gid not found') 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: uid/gid not found 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:22:42.889801+0000 mgr.smithi038.powepi (mgr.14156) 162 : cephadm [INF] Deploying daemon crash.smithi038 on smithi038 2024-02-20T13:22:44.454 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:43.274819+0000 mon.smithi038 (mon.0) 195 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:22:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 -- 172.21.15.114:0/1934237413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 msgr2=0x7f19d40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 --2- 172.21.15.114:0/1934237413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d40f58a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f19c4009fb0 tx=0x7f19c4009b30).stop 2024-02-20T13:22:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 -- 172.21.15.114:0/1934237413 shutdown_connections 2024-02-20T13:22:45.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 --2- 172.21.15.114:0/1934237413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d40f58a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 -- 172.21.15.114:0/1934237413 >> 172.21.15.114:0/1934237413 conn(0x7f19d40f0be0 msgr2=0x7f19d40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 -- 172.21.15.114:0/1934237413 shutdown_connections 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 -- 172.21.15.114:0/1934237413 wait complete. 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 Processor -- start 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.332+0000 7f19dbb5a700 1 -- start start 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19dbb5a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d4109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19dbb5a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d4060300 con 0x7f19d40f5480 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19d98f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d4109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19d98f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d4109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:57008/0 (socket says 172.21.15.114:57008) 2024-02-20T13:22:45.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19d98f6700 1 -- 172.21.15.114:0/2352566908 learned_addr learned my addr 172.21.15.114:0/2352566908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19d98f6700 1 -- 172.21.15.114:0/2352566908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4005040 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19d98f6700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d4109510 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f19c40089f0 tx=0x7f19c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19c4009320 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d4060500 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f19c4013ec0 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19c4014d30 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.333+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d4109bc0 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.334+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f19c40095f0 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.334+0000 7f19ca7fc700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f19c00310e0 0x7f19c00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.334+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f19c4042450 con 0x7f19d40f5480 2024-02-20T13:22:45.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.334+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d40591e0 con 0x7f19d40f5480 2024-02-20T13:22:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.334+0000 7f19d90f5700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f19c00310e0 0x7f19c00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.335+0000 7f19d90f5700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f19c00310e0 0x7f19c00335a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f19d00096a0 tx=0x7f19d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:45.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.341+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f19c401dd70 con 0x7f19d40f5480 2024-02-20T13:22:45.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:44.184806+0000 mgr.smithi038.powepi (mgr.14156) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.548+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19d4045bb0 con 0x7f19d40f5480 2024-02-20T13:22:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.549+0000 7f19ca7fc700 1 -- 172.21.15.114:0/2352566908 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f19c402b020 con 0x7f19d40f5480 2024-02-20T13:22:45.550 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:45.550 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f19c00310e0 msgr2=0x7f19c00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f19c00310e0 0x7f19c00335a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f19d00096a0 tx=0x7f19d0006b40).stop 2024-02-20T13:22:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 msgr2=0x7f19d4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d4109510 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f19c40089f0 tx=0x7f19c400b040).stop 2024-02-20T13:22:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 shutdown_connections 2024-02-20T13:22:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f19c00310e0 0x7f19c00335a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 --2- 172.21.15.114:0/2352566908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d40f5480 0x7f19d4109510 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 >> 172.21.15.114:0/2352566908 conn(0x7f19d40f0be0 msgr2=0x7f19d40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 shutdown_connections 2024-02-20T13:22:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:45.550+0000 7f19dbb5a700 1 -- 172.21.15.114:0/2352566908 wait complete. 2024-02-20T13:22:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.362718+0000 mon.smithi038 (mon.0) 196 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:22:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.422537+0000 mon.smithi038 (mon.0) 197 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:22:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.424761+0000 mon.smithi038 (mon.0) 198 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:22:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.425159+0000 mon.smithi038 (mon.0) 199 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T13:22:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.427953+0000 mon.smithi038 (mon.0) 200 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:22:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.549452+0000 mon.smithi038 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.114:0/2352566908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:47.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:47.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:47.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:45.425518+0000 mgr.smithi038.powepi (mgr.14156) 164 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T13:22:47.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:22:45.431196+0000 mgr.smithi038.powepi (mgr.14156) 165 : cephadm [INF] Deploying daemon grafana.smithi038 on smithi038 2024-02-20T13:22:47.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:46.185219+0000 mgr.smithi038.powepi (mgr.14156) 166 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 -- 172.21.15.114:0/2068649254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 msgr2=0x7fce940f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 --2- 172.21.15.114:0/2068649254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940f6d70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fce88009fb0 tx=0x7fce88009b30).stop 2024-02-20T13:22:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 -- 172.21.15.114:0/2068649254 shutdown_connections 2024-02-20T13:22:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 --2- 172.21.15.114:0/2068649254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940f6d70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 -- 172.21.15.114:0/2068649254 >> 172.21.15.114:0/2068649254 conn(0x7fce940f0190 msgr2=0x7fce940f25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 -- 172.21.15.114:0/2068649254 shutdown_connections 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.215+0000 7fce9a79d700 1 -- 172.21.15.114:0/2068649254 wait complete. 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9a79d700 1 Processor -- start 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9a79d700 1 -- start start 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9a79d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940fb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9a79d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce940f99c0 con 0x7fce940f6950 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9979b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940fb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9979b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940fb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:57018/0 (socket says 172.21.15.114:57018) 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9979b700 1 -- 172.21.15.114:0/1119122248 learned_addr learned my addr 172.21.15.114:0/1119122248 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9979b700 1 -- 172.21.15.114:0/1119122248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce88005040 con 0x7fce940f6950 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9979b700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940fb540 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fce88009f80 tx=0x7fce8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce88009320 con 0x7fce940f6950 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce940f9b60 con 0x7fce940f6950 2024-02-20T13:22:49.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fce88013680 con 0x7fce940f6950 2024-02-20T13:22:49.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.216+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce8801eda0 con 0x7fce940f6950 2024-02-20T13:22:49.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.217+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce940f9ff0 con 0x7fce940f6950 2024-02-20T13:22:49.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.217+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fce88014420 con 0x7fce940f6950 2024-02-20T13:22:49.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.218+0000 7fce85ffb700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fce80031130 0x7fce800335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:49.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.218+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fce880423c0 con 0x7fce940f6950 2024-02-20T13:22:49.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.218+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce940591e0 con 0x7fce940f6950 2024-02-20T13:22:49.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.218+0000 7fce98f9a700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fce80031130 0x7fce800335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:49.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.218+0000 7fce98f9a700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fce80031130 0x7fce800335f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fce900096a0 tx=0x7fce90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:49.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.223+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fce880146b0 con 0x7fce940f6950 2024-02-20T13:22:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.428+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce940fa6f0 con 0x7fce940f6950 2024-02-20T13:22:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.429+0000 7fce85ffb700 1 -- 172.21.15.114:0/1119122248 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce88026350 con 0x7fce940f6950 2024-02-20T13:22:49.429 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:49.430 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fce80031130 msgr2=0x7fce800335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fce80031130 0x7fce800335f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fce900096a0 tx=0x7fce90006b40).stop 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 msgr2=0x7fce940fb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940fb540 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fce88009f80 tx=0x7fce8800b040).stop 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 shutdown_connections 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fce80031130 0x7fce800335f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 --2- 172.21.15.114:0/1119122248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce940f6950 0x7fce940fb540 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 >> 172.21.15.114:0/1119122248 conn(0x7fce940f0190 msgr2=0x7fce940f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 shutdown_connections 2024-02-20T13:22:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:49.430+0000 7fce9a79d700 1 -- 172.21.15.114:0/1119122248 wait complete. 2024-02-20T13:22:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:48.185557+0000 mgr.smithi038.powepi (mgr.14156) 167 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:48.279911+0000 mon.smithi038 (mon.0) 202 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:22:50.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:50 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:49.429499+0000 mon.smithi038 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.114:0/1119122248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:51.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:51.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:51.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:50.185868+0000 mgr.smithi038.powepi (mgr.14156) 168 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 -- 172.21.15.114:0/44327890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 msgr2=0x7f35f80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 --2- 172.21.15.114:0/44327890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80f58a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f35ec009fb0 tx=0x7f35ec009b30).stop 2024-02-20T13:22:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 -- 172.21.15.114:0/44327890 shutdown_connections 2024-02-20T13:22:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 --2- 172.21.15.114:0/44327890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80f58a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 -- 172.21.15.114:0/44327890 >> 172.21.15.114:0/44327890 conn(0x7f35f80f0be0 msgr2=0x7f35f80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 -- 172.21.15.114:0/44327890 shutdown_connections 2024-02-20T13:22:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.876+0000 7f35fe1e3700 1 -- 172.21.15.114:0/44327890 wait complete. 2024-02-20T13:22:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35fe1e3700 1 Processor -- start 2024-02-20T13:22:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35fe1e3700 1 -- start start 2024-02-20T13:22:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35fe1e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35fe1e3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80fb8e0 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35f77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35f77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:57036/0 (socket says 172.21.15.114:57036) 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35f77fe700 1 -- 172.21.15.114:0/2254047740 learned_addr learned my addr 172.21.15.114:0/2254047740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.877+0000 7f35f77fe700 1 -- 172.21.15.114:0/2254047740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec005040 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.878+0000 7f35f77fe700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80fd460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f35ec0089f0 tx=0x7f35ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.878+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ec009320 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.878+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f80fba80 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.878+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f35ec013ec0 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.878+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ec014d30 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.878+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f80fbf10 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.879+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f35ec014420 con 0x7f35f80f5480 2024-02-20T13:22:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.879+0000 7f35dffff700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f35e0031090 0x7f35e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.879+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f80591e0 con 0x7f35f80f5480 2024-02-20T13:22:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.879+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f35ec0424c0 con 0x7f35f80f5480 2024-02-20T13:22:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.879+0000 7f35f6ffd700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f35e0031090 0x7f35e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.879+0000 7f35f6ffd700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f35e0031090 0x7f35e0033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f35e80096a0 tx=0x7f35e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:52.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:52.886+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f35ec014750 con 0x7f35f80f5480 2024-02-20T13:22:53.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.093+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35f8045bb0 con 0x7f35f80f5480 2024-02-20T13:22:53.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.094+0000 7f35dffff700 1 -- 172.21.15.114:0/2254047740 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35ec01d380 con 0x7f35f80f5480 2024-02-20T13:22:53.095 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:53.095 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f35e0031090 msgr2=0x7f35e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f35e0031090 0x7f35e0033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f35e80096a0 tx=0x7f35e8006b40).stop 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 msgr2=0x7f35f80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80fd460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f35ec0089f0 tx=0x7f35ec00b040).stop 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 shutdown_connections 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f35e0031090 0x7f35e0033550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 --2- 172.21.15.114:0/2254047740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f80f5480 0x7f35f80fd460 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 >> 172.21.15.114:0/2254047740 conn(0x7f35f80f0be0 msgr2=0x7f35f805ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:53.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 shutdown_connections 2024-02-20T13:22:53.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:53.095+0000 7f35fe1e3700 1 -- 172.21.15.114:0/2254047740 wait complete. 2024-02-20T13:22:53.097 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:53.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:52.186190+0000 mgr.smithi038.powepi (mgr.14156) 169 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:53.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:53.094537+0000 mon.smithi038 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.114:0/2254047740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:54.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:54.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:55.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:54.186396+0000 mgr.smithi038.powepi (mgr.14156) 170 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.741+0000 7ff7f34d2700 1 -- 172.21.15.114:0/2494463297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 msgr2=0x7ff7ec0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.741+0000 7ff7f34d2700 1 --2- 172.21.15.114:0/2494463297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec0f58a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff7dc009fb0 tx=0x7ff7dc009b30).stop 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.741+0000 7ff7f34d2700 1 -- 172.21.15.114:0/2494463297 shutdown_connections 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.741+0000 7ff7f34d2700 1 --2- 172.21.15.114:0/2494463297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec0f58a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.741+0000 7ff7f34d2700 1 -- 172.21.15.114:0/2494463297 >> 172.21.15.114:0/2494463297 conn(0x7ff7ec0f0be0 msgr2=0x7ff7ec0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f34d2700 1 -- 172.21.15.114:0/2494463297 shutdown_connections 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f34d2700 1 -- 172.21.15.114:0/2494463297 wait complete. 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f34d2700 1 Processor -- start 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f34d2700 1 -- start start 2024-02-20T13:22:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f34d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f34d2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec105be0 con 0x7ff7ec0f5480 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f126e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f126e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41094/0 (socket says 172.21.15.114:41094) 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.742+0000 7ff7f126e700 1 -- 172.21.15.114:0/1454526411 learned_addr learned my addr 172.21.15.114:0/1454526411 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7f126e700 1 -- 172.21.15.114:0/1454526411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc005040 con 0x7ff7ec0f5480 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7f126e700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec1056f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ff7dc000c00 tx=0x7ff7dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7dc0095f0 con 0x7ff7ec0f5480 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ec105de0 con 0x7ff7ec0f5480 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec106270 con 0x7ff7ec0f5480 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff7dc007e50 con 0x7ff7ec0f5480 2024-02-20T13:22:56.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.743+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7dc014e00 con 0x7ff7ec0f5480 2024-02-20T13:22:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.744+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff7dc025070 con 0x7ff7ec0f5480 2024-02-20T13:22:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.744+0000 7ff7e1ffb700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7d8031090 0x7ff7d8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:22:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.744+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff7dc041d10 con 0x7ff7ec0f5480 2024-02-20T13:22:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.744+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec045bb0 con 0x7ff7ec0f5480 2024-02-20T13:22:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.744+0000 7ff7f0a6d700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7d8031090 0x7ff7d8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:22:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.745+0000 7ff7f0a6d700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7d8031090 0x7ff7d8033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff7e80096a0 tx=0x7ff7e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:22:56.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.750+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff7dc0079c0 con 0x7ff7ec0f5480 2024-02-20T13:22:56.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.961+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7ec106ef0 con 0x7ff7ec0f5480 2024-02-20T13:22:56.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.961+0000 7ff7e1ffb700 1 -- 172.21.15.114:0/1454526411 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7dc00f300 con 0x7ff7ec0f5480 2024-02-20T13:22:56.962 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:22:56.962 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7d8031090 msgr2=0x7ff7d8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7d8031090 0x7ff7d8033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff7e80096a0 tx=0x7ff7e8006b40).stop 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 msgr2=0x7ff7ec1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec1056f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ff7dc000c00 tx=0x7ff7dc00b040).stop 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 shutdown_connections 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7d8031090 0x7ff7d8033550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 --2- 172.21.15.114:0/1454526411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7ec0f5480 0x7ff7ec1056f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 >> 172.21.15.114:0/1454526411 conn(0x7ff7ec0f0be0 msgr2=0x7ff7ec0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 shutdown_connections 2024-02-20T13:22:56.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:22:56.963+0000 7ff7f34d2700 1 -- 172.21.15.114:0/1454526411 wait complete. 2024-02-20T13:22:56.965 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:22:57.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:56.186632+0000 mgr.smithi038.powepi (mgr.14156) 171 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:22:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:22:56.962079+0000 mon.smithi038 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.114:0/1454526411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:22:58.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:22:58.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:22:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:22:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:22:58.186884+0000 mgr.smithi038.powepi (mgr.14156) 172 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:00.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 -- 172.21.15.114:0/322466035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 msgr2=0x7f69100f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:00.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 --2- 172.21.15.114:0/322466035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f69100f56f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f6904009fb0 tx=0x7f6904009b30).stop 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 -- 172.21.15.114:0/322466035 shutdown_connections 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 --2- 172.21.15.114:0/322466035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f69100f56f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 -- 172.21.15.114:0/322466035 >> 172.21.15.114:0/322466035 conn(0x7f69100f0a30 msgr2=0x7f69100f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 -- 172.21.15.114:0/322466035 shutdown_connections 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 -- 172.21.15.114:0/322466035 wait complete. 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 Processor -- start 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.670+0000 7f69158da700 1 -- start start 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69158da700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f6910109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69158da700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910060320 con 0x7f69100f52d0 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69148d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f6910109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69148d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f6910109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41110/0 (socket says 172.21.15.114:41110) 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69148d8700 1 -- 172.21.15.114:0/1539910368 learned_addr learned my addr 172.21.15.114:0/1539910368 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:00.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69148d8700 1 -- 172.21.15.114:0/1539910368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6904005040 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.671+0000 7f69148d8700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f6910109310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f69040089f0 tx=0x7f690400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6904009320 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6904013ec0 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6910060520 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6904014d30 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69101099c0 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f69040095f0 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.672+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6910045bb0 con 0x7f69100f52d0 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.673+0000 7f690d7fa700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6900031090 0x7f6900033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:00.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.673+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f69040423e0 con 0x7f69100f52d0 2024-02-20T13:23:00.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.673+0000 7f690ffff700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6900031090 0x7f6900033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:00.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.673+0000 7f690ffff700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6900031090 0x7f6900033550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f68fc0096a0 tx=0x7f68fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:00.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.677+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f690401d980 con 0x7f69100f52d0 2024-02-20T13:23:00.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.886+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f691010a1e0 con 0x7f69100f52d0 2024-02-20T13:23:00.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.887+0000 7f690d7fa700 1 -- 172.21.15.114:0/1539910368 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f690401d380 con 0x7f69100f52d0 2024-02-20T13:23:00.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:00.888 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:00.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.888+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6900031090 msgr2=0x7f6900033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:00.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.888+0000 7f69158da700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6900031090 0x7f6900033550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f68fc0096a0 tx=0x7f68fc006b40).stop 2024-02-20T13:23:00.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 msgr2=0x7f6910109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f6910109310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f69040089f0 tx=0x7f690400b040).stop 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 shutdown_connections 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6900031090 0x7f6900033550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 --2- 172.21.15.114:0/1539910368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69100f52d0 0x7f6910109310 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 >> 172.21.15.114:0/1539910368 conn(0x7f69100f0a30 msgr2=0x7f69100f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 shutdown_connections 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:00.889+0000 7f69158da700 1 -- 172.21.15.114:0/1539910368 wait complete. 2024-02-20T13:23:00.890 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:01.257 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:00.187177+0000 mgr.smithi038.powepi (mgr.14156) 173 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:01.257 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:00.887426+0000 mon.smithi038 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.114:0/1539910368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:02.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:02.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:03.327 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:02.187577+0000 mgr.smithi038.powepi (mgr.14156) 174 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:03.327 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:03.082326+0000 mon.smithi038 (mon.0) 207 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:23:04.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- 172.21.15.114:0/1361627661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 msgr2=0x7fb27c0f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 --2- 172.21.15.114:0/1361627661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0f5700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fb278009fb0 tx=0x7fb278009b30).stop 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- 172.21.15.114:0/1361627661 shutdown_connections 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 --2- 172.21.15.114:0/1361627661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0f5700 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- 172.21.15.114:0/1361627661 >> 172.21.15.114:0/1361627661 conn(0x7fb27c0f0a40 msgr2=0x7fb27c0f2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- 172.21.15.114:0/1361627661 shutdown_connections 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- 172.21.15.114:0/1361627661 wait complete. 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 Processor -- start 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- start start 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0fd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.634+0000 7fb283035700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0fb740 con 0x7fb27c0f52e0 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.635+0000 7fb282033700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0fd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.635+0000 7fb282033700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0fd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41116/0 (socket says 172.21.15.114:41116) 2024-02-20T13:23:04.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.635+0000 7fb282033700 1 -- 172.21.15.114:0/578594795 learned_addr learned my addr 172.21.15.114:0/578594795 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.635+0000 7fb282033700 1 -- 172.21.15.114:0/578594795 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb278005040 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.635+0000 7fb282033700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0fd2c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb2780089f0 tx=0x7fb27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb278009320 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c0fb8e0 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb278013ec0 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb278014d30 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c0fbd70 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb2780095f0 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.636+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c0591e0 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.637+0000 7fb272ffd700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb2680310e0 0x7fb2680335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.637+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb278042390 con 0x7fb27c0f52e0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.637+0000 7fb281832700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb2680310e0 0x7fb2680335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.637+0000 7fb281832700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb2680310e0 0x7fb2680335a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb26c0096a0 tx=0x7fb26c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.642+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb27801d980 con 0x7fb27c0f52e0 2024-02-20T13:23:04.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:23:03.084203+0000 mgr.smithi038.powepi (mgr.14156) 175 : cephadm [INF] Deploying daemon node-exporter.smithi038 on smithi038 2024-02-20T13:23:04.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:03.282980+0000 mon.smithi038 (mon.0) 208 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:23:04.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.851+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb27c0fadd0 con 0x7fb27c0f52e0 2024-02-20T13:23:04.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.852+0000 7fb272ffd700 1 -- 172.21.15.114:0/578594795 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb27801d380 con 0x7fb27c0f52e0 2024-02-20T13:23:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:04.853 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb2680310e0 msgr2=0x7fb2680335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb2680310e0 0x7fb2680335a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb26c0096a0 tx=0x7fb26c006b40).stop 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 msgr2=0x7fb27c0fd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0fd2c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb2780089f0 tx=0x7fb27800b040).stop 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 shutdown_connections 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb2680310e0 0x7fb2680335a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 --2- 172.21.15.114:0/578594795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb27c0f52e0 0x7fb27c0fd2c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 >> 172.21.15.114:0/578594795 conn(0x7fb27c0f0a40 msgr2=0x7fb27c05fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 shutdown_connections 2024-02-20T13:23:04.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:04.854+0000 7fb283035700 1 -- 172.21.15.114:0/578594795 wait complete. 2024-02-20T13:23:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:05.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:04.187907+0000 mgr.smithi038.powepi (mgr.14156) 176 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:05.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:04.852823+0000 mon.smithi038 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.114:0/578594795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:06.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:06.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:07.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:06.188278+0000 mgr.smithi038.powepi (mgr.14156) 177 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 -- 172.21.15.114:0/1174322851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 msgr2=0x7f10e80f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 --2- 172.21.15.114:0/1174322851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e80f6800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f10d8009fb0 tx=0x7f10d8009b30).stop 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 -- 172.21.15.114:0/1174322851 shutdown_connections 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 --2- 172.21.15.114:0/1174322851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e80f6800 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 -- 172.21.15.114:0/1174322851 >> 172.21.15.114:0/1174322851 conn(0x7f10e80f1b40 msgr2=0x7f10e80f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 -- 172.21.15.114:0/1174322851 shutdown_connections 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 -- 172.21.15.114:0/1174322851 wait complete. 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 Processor -- start 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.659+0000 7f10e759e700 1 -- start start 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e759e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e8106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e759e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e8106b10 con 0x7f10e80f63e0 2024-02-20T13:23:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e8106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e8106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41762/0 (socket says 172.21.15.114:41762) 2024-02-20T13:23:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e659c700 1 -- 172.21.15.114:0/1839889195 learned_addr learned my addr 172.21.15.114:0/1839889195 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e659c700 1 -- 172.21.15.114:0/1839889195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8005040 con 0x7f10e80f63e0 2024-02-20T13:23:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10e659c700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e8106620 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f10d80089f0 tx=0x7f10d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.660+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10d8009320 con 0x7f10e80f63e0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.661+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e8106d10 con 0x7f10e80f63e0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.661+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f10d8013ec0 con 0x7f10e80f63e0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.661+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10d8014d30 con 0x7f10e80f63e0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.661+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e81071a0 con 0x7f10e80f63e0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.661+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f10d80263a0 con 0x7f10e80f63e0 2024-02-20T13:23:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.661+0000 7f10d6ffd700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f10d0031090 0x7f10d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.662+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f10d8042480 con 0x7f10e80f63e0 2024-02-20T13:23:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.662+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e8045bb0 con 0x7f10e80f63e0 2024-02-20T13:23:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.662+0000 7f10e5d9b700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f10d0031090 0x7f10d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.662+0000 7f10e5d9b700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f10d0031090 0x7f10d0033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f10dc0096a0 tx=0x7f10dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:08.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.666+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f10d80266d0 con 0x7f10e80f63e0 2024-02-20T13:23:08.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.871+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10e8107e20 con 0x7f10e80f63e0 2024-02-20T13:23:08.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.872+0000 7f10d6ffd700 1 -- 172.21.15.114:0/1839889195 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10d802b020 con 0x7f10e80f63e0 2024-02-20T13:23:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:08.874 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:08.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f10d0031090 msgr2=0x7f10d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:08.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f10d0031090 0x7f10d0033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f10dc0096a0 tx=0x7f10dc006b40).stop 2024-02-20T13:23:08.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 msgr2=0x7f10e8106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e8106620 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f10d80089f0 tx=0x7f10d800b040).stop 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 shutdown_connections 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f10d0031090 0x7f10d0033550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 --2- 172.21.15.114:0/1839889195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80f63e0 0x7f10e8106620 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 >> 172.21.15.114:0/1839889195 conn(0x7f10e80f1b40 msgr2=0x7f10e80f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 shutdown_connections 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:08.874+0000 7f10e759e700 1 -- 172.21.15.114:0/1839889195 wait complete. 2024-02-20T13:23:08.876 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:09.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:08.188632+0000 mgr.smithi038.powepi (mgr.14156) 178 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:08.873849+0000 mon.smithi038 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.114:0/1839889195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:10.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:10.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:11.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:10.189022+0000 mgr.smithi038.powepi (mgr.14156) 179 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:12.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 -- 172.21.15.114:0/2354798786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 msgr2=0x7fcfc40f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:12.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 --2- 172.21.15.114:0/2354798786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc40f4c70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fcfb8009fb0 tx=0x7fcfb8009b30).stop 2024-02-20T13:23:12.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 -- 172.21.15.114:0/2354798786 shutdown_connections 2024-02-20T13:23:12.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 --2- 172.21.15.114:0/2354798786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc40f4c70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:12.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 -- 172.21.15.114:0/2354798786 >> 172.21.15.114:0/2354798786 conn(0x7fcfc40f0390 msgr2=0x7fcfc40f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 -- 172.21.15.114:0/2354798786 shutdown_connections 2024-02-20T13:23:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.508+0000 7fcfc959d700 1 -- 172.21.15.114:0/2354798786 wait complete. 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc959d700 1 Processor -- start 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc959d700 1 -- start start 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc959d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc959d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4106c70 con 0x7fcfc40f4850 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41790/0 (socket says 172.21.15.114:41790) 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.509+0000 7fcfc2ffd700 1 -- 172.21.15.114:0/4056150923 learned_addr learned my addr 172.21.15.114:0/4056150923 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfc2ffd700 1 -- 172.21.15.114:0/4056150923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb8005040 con 0x7fcfc40f4850 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfc2ffd700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc4106780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fcfb80089f0 tx=0x7fcfb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb8009320 con 0x7fcfc40f4850 2024-02-20T13:23:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcfb8013ec0 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc4106e70 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb8014d30 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.510+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc4107300 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.511+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcfb8014420 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.511+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc4045bb0 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.511+0000 7fcfab7fe700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcfac031090 0x7fcfac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.511+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcfb80424c0 con 0x7fcfc40f4850 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.511+0000 7fcfc27fc700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcfac031090 0x7fcfac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:12.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.512+0000 7fcfc27fc700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcfac031090 0x7fcfac033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcfb40096a0 tx=0x7fcfb4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:12.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.516+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcfb801d980 con 0x7fcfc40f4850 2024-02-20T13:23:12.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.723+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfc41081b0 con 0x7fcfc40f4850 2024-02-20T13:23:12.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.724+0000 7fcfab7fe700 1 -- 172.21.15.114:0/4056150923 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfb801d380 con 0x7fcfc40f4850 2024-02-20T13:23:12.728 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.725+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcfac031090 msgr2=0x7fcfac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.725+0000 7fcfc959d700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcfac031090 0x7fcfac033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcfb40096a0 tx=0x7fcfb4006b40).stop 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.725+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 msgr2=0x7fcfc4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.725+0000 7fcfc959d700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc4106780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fcfb80089f0 tx=0x7fcfb800b040).stop 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.726+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 shutdown_connections 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.726+0000 7fcfc959d700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcfac031090 0x7fcfac033550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.726+0000 7fcfc959d700 1 --2- 172.21.15.114:0/4056150923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc40f4850 0x7fcfc4106780 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:12.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.726+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 >> 172.21.15.114:0/4056150923 conn(0x7fcfc40f0390 msgr2=0x7fcfc40f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.726+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 shutdown_connections 2024-02-20T13:23:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:12.726+0000 7fcfc959d700 1 -- 172.21.15.114:0/4056150923 wait complete. 2024-02-20T13:23:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:13.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:12.189417+0000 mgr.smithi038.powepi (mgr.14156) 180 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:13.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:12.725590+0000 mon.smithi038 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.114:0/4056150923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:14.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:14.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:15.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:14.189698+0000 mgr.smithi038.powepi (mgr.14156) 181 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 -- 172.21.15.114:0/670477305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 msgr2=0x7f54800f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 --2- 172.21.15.114:0/670477305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54800f58a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f5468009fb0 tx=0x7f5468009b30).stop 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 -- 172.21.15.114:0/670477305 shutdown_connections 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 --2- 172.21.15.114:0/670477305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54800f58a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 -- 172.21.15.114:0/670477305 >> 172.21.15.114:0/670477305 conn(0x7f54800f0be0 msgr2=0x7f54800f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 -- 172.21.15.114:0/670477305 shutdown_connections 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 -- 172.21.15.114:0/670477305 wait complete. 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.671+0000 7f54864eb700 1 Processor -- start 2024-02-20T13:23:16.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f54864eb700 1 -- start start 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f54864eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54801056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f54864eb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5480105be0 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f547ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54801056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f547ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54801056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35524/0 (socket says 172.21.15.114:35524) 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f547ffff700 1 -- 172.21.15.114:0/3297140371 learned_addr learned my addr 172.21.15.114:0/3297140371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.672+0000 7f547ffff700 1 -- 172.21.15.114:0/3297140371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5468005040 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.673+0000 7f547ffff700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54801056f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f54680089f0 tx=0x7f546800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.673+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5468009320 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.673+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5480105de0 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.673+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5468013ec0 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.673+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5468014d30 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.673+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5480106270 con 0x7f54800f5480 2024-02-20T13:23:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.674+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f54680095f0 con 0x7f54800f5480 2024-02-20T13:23:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.674+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5480045bb0 con 0x7f54800f5480 2024-02-20T13:23:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.674+0000 7f547cff9700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f546c031090 0x7f546c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.674+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f54680424a0 con 0x7f54800f5480 2024-02-20T13:23:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.674+0000 7f547f7fe700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f546c031090 0x7f546c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.674+0000 7f547f7fe700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f546c031090 0x7f546c033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f54700096a0 tx=0x7f5470006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.679+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f546801d980 con 0x7f54800f5480 2024-02-20T13:23:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.885+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54800faf70 con 0x7f54800f5480 2024-02-20T13:23:16.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.885+0000 7f547cff9700 1 -- 172.21.15.114:0/3297140371 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f546801d380 con 0x7f54800f5480 2024-02-20T13:23:16.887 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:16.887 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f546c031090 msgr2=0x7f546c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f546c031090 0x7f546c033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f54700096a0 tx=0x7f5470006b40).stop 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 msgr2=0x7f54801056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54801056f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f54680089f0 tx=0x7f546800b040).stop 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 shutdown_connections 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f546c031090 0x7f546c033550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 --2- 172.21.15.114:0/3297140371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54800f5480 0x7f54801056f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 >> 172.21.15.114:0/3297140371 conn(0x7f54800f0be0 msgr2=0x7f54800f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 shutdown_connections 2024-02-20T13:23:16.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:16.887+0000 7f54864eb700 1 -- 172.21.15.114:0/3297140371 wait complete. 2024-02-20T13:23:16.890 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:17.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:16.190057+0000 mgr.smithi038.powepi (mgr.14156) 182 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:17.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:16.887174+0000 mon.smithi038 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.114:0/3297140371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:18.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:18.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:19.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:18.190431+0000 mgr.smithi038.powepi (mgr.14156) 183 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:20.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 -- 172.21.15.114:0/515380792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 msgr2=0x7f1c380f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 --2- 172.21.15.114:0/515380792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c380f58a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f1c28009fb0 tx=0x7f1c28009b30).stop 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 -- 172.21.15.114:0/515380792 shutdown_connections 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 --2- 172.21.15.114:0/515380792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c380f58a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 -- 172.21.15.114:0/515380792 >> 172.21.15.114:0/515380792 conn(0x7f1c380f0be0 msgr2=0x7f1c380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 -- 172.21.15.114:0/515380792 shutdown_connections 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.676+0000 7f1c3ea29700 1 -- 172.21.15.114:0/515380792 wait complete. 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c3ea29700 1 Processor -- start 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c3ea29700 1 -- start start 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c3ea29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c381067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:20.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c3ea29700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c38106cc0 con 0x7f1c380f5480 2024-02-20T13:23:20.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c37fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c381067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:20.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c37fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c381067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35550/0 (socket says 172.21.15.114:35550) 2024-02-20T13:23:20.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c37fff700 1 -- 172.21.15.114:0/1892106531 learned_addr learned my addr 172.21.15.114:0/1892106531 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:20.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.677+0000 7f1c37fff700 1 -- 172.21.15.114:0/1892106531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28005040 con 0x7f1c380f5480 2024-02-20T13:23:20.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.678+0000 7f1c37fff700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c381067d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1c28000c00 tx=0x7f1c2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.678+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c280095f0 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.678+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c38106ec0 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.678+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1c28007e50 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.678+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c28014e70 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.678+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c38107350 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.679+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1c28025070 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.679+0000 7f1c34ff9700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c20031090 0x7f1c20033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.679+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c38045bb0 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.679+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1c28041e10 con 0x7f1c380f5480 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.679+0000 7f1c377fe700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c20031090 0x7f1c20033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:20.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.680+0000 7f1c377fe700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c20031090 0x7f1c20033550 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1c2c0096a0 tx=0x7f1c2c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:20.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.684+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1c280146b0 con 0x7f1c380f5480 2024-02-20T13:23:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.894+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c381081c0 con 0x7f1c380f5480 2024-02-20T13:23:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.895+0000 7f1c34ff9700 1 -- 172.21.15.114:0/1892106531 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c280146b0 con 0x7f1c380f5480 2024-02-20T13:23:20.897 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:20.897 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.896+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c20031090 msgr2=0x7f1c20033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c20031090 0x7f1c20033550 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1c2c0096a0 tx=0x7f1c2c006b40).stop 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 msgr2=0x7f1c381067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c381067d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1c28000c00 tx=0x7f1c2800b040).stop 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 shutdown_connections 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c20031090 0x7f1c20033550 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 --2- 172.21.15.114:0/1892106531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c380f5480 0x7f1c381067d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 >> 172.21.15.114:0/1892106531 conn(0x7f1c380f0be0 msgr2=0x7f1c3805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 shutdown_connections 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:20.897+0000 7f1c3ea29700 1 -- 172.21.15.114:0/1892106531 wait complete. 2024-02-20T13:23:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:21.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:20.190800+0000 mgr.smithi038.powepi (mgr.14156) 184 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:21.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:20.896460+0000 mon.smithi038 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.114:0/1892106531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:22.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:22.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:23.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:22.191137+0000 mgr.smithi038.powepi (mgr.14156) 185 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:24.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 -- 172.21.15.114:0/1097762265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 msgr2=0x7fed400f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 --2- 172.21.15.114:0/1097762265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed400f56f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fed34009fb0 tx=0x7fed34009b30).stop 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 -- 172.21.15.114:0/1097762265 shutdown_connections 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 --2- 172.21.15.114:0/1097762265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed400f56f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 -- 172.21.15.114:0/1097762265 >> 172.21.15.114:0/1097762265 conn(0x7fed400f0a30 msgr2=0x7fed400f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 -- 172.21.15.114:0/1097762265 shutdown_connections 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 -- 172.21.15.114:0/1097762265 wait complete. 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.692+0000 7fed455a8700 1 Processor -- start 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed455a8700 1 -- start start 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed455a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed40105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed455a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed40105a00 con 0x7fed400f52d0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed3ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed40105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed3ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed40105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35568/0 (socket says 172.21.15.114:35568) 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed3ffff700 1 -- 172.21.15.114:0/2969585983 learned_addr learned my addr 172.21.15.114:0/2969585983 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.693+0000 7fed3ffff700 1 -- 172.21.15.114:0/2969585983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34005040 con 0x7fed400f52d0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.694+0000 7fed3ffff700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed40105510 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fed340089f0 tx=0x7fed3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.694+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed34009320 con 0x7fed400f52d0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.694+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fed34013ec0 con 0x7fed400f52d0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.694+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed40105c00 con 0x7fed400f52d0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.694+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed34014d30 con 0x7fed400f52d0 2024-02-20T13:23:24.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.694+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed40106090 con 0x7fed400f52d0 2024-02-20T13:23:24.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.695+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fed340095f0 con 0x7fed400f52d0 2024-02-20T13:23:24.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.695+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed40045bb0 con 0x7fed400f52d0 2024-02-20T13:23:24.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.695+0000 7fed3cff9700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fed28031090 0x7fed28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:24.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.695+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fed340423e0 con 0x7fed400f52d0 2024-02-20T13:23:24.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.695+0000 7fed3f7fe700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fed28031090 0x7fed28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:24.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.696+0000 7fed3f7fe700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fed28031090 0x7fed28033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fed300096a0 tx=0x7fed30006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.700+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fed3401d7f0 con 0x7fed400f52d0 2024-02-20T13:23:24.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.909+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed40106f40 con 0x7fed400f52d0 2024-02-20T13:23:24.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.909+0000 7fed3cff9700 1 -- 172.21.15.114:0/2969585983 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed3401d7f0 con 0x7fed400f52d0 2024-02-20T13:23:24.911 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:24.911 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fed28031090 msgr2=0x7fed28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fed28031090 0x7fed28033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fed300096a0 tx=0x7fed30006b40).stop 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 msgr2=0x7fed40105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed40105510 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fed340089f0 tx=0x7fed3400b040).stop 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 shutdown_connections 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fed28031090 0x7fed28033550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 --2- 172.21.15.114:0/2969585983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed400f52d0 0x7fed40105510 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:24.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 >> 172.21.15.114:0/2969585983 conn(0x7fed400f0a30 msgr2=0x7fed400f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 shutdown_connections 2024-02-20T13:23:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:24.911+0000 7fed455a8700 1 -- 172.21.15.114:0/2969585983 wait complete. 2024-02-20T13:23:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:24.191433+0000 mgr.smithi038.powepi (mgr.14156) 186 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:24.910980+0000 mon.smithi038 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.114:0/2969585983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:26.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:26.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:27.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:26.191703+0000 mgr.smithi038.powepi (mgr.14156) 187 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 -- 172.21.15.114:0/648470734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 msgr2=0x7f3fcc0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 --2- 172.21.15.114:0/648470734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc0f9160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3fbc009fb0 tx=0x7f3fbc009b30).stop 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 -- 172.21.15.114:0/648470734 shutdown_connections 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 --2- 172.21.15.114:0/648470734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc0f9160 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 -- 172.21.15.114:0/648470734 >> 172.21.15.114:0/648470734 conn(0x7f3fcc0f0be0 msgr2=0x7f3fcc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 -- 172.21.15.114:0/648470734 shutdown_connections 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.665+0000 7f3fd4121700 1 -- 172.21.15.114:0/648470734 wait complete. 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd4121700 1 Processor -- start 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd4121700 1 -- start start 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd4121700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:28.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd4121700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc106c40 con 0x7f3fcc060360 2024-02-20T13:23:28.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd1ebd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:28.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd1ebd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34508/0 (socket says 172.21.15.114:34508) 2024-02-20T13:23:28.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd1ebd700 1 -- 172.21.15.114:0/2930215150 learned_addr learned my addr 172.21.15.114:0/2930215150 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.666+0000 7f3fd1ebd700 1 -- 172.21.15.114:0/2930215150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc005040 con 0x7f3fcc060360 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.667+0000 7f3fd1ebd700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc106750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f3fbc009f80 tx=0x7f3fbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.667+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fbc009320 con 0x7f3fcc060360 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.667+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc106e40 con 0x7f3fcc060360 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.667+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fbc013680 con 0x7f3fcc060360 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.667+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fbc01eda0 con 0x7f3fcc060360 2024-02-20T13:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.667+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc1072d0 con 0x7f3fcc060360 2024-02-20T13:23:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.668+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3fbc014420 con 0x7f3fcc060360 2024-02-20T13:23:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.668+0000 7f3fc27fc700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fb8031090 0x7f3fb8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.668+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fbc042360 con 0x7f3fcc060360 2024-02-20T13:23:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.668+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc045bb0 con 0x7f3fcc060360 2024-02-20T13:23:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.668+0000 7f3fd16bc700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fb8031090 0x7f3fb8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.669+0000 7f3fd16bc700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fb8031090 0x7f3fb8033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3fc80096a0 tx=0x7f3fc8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:28.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.673+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fbc014700 con 0x7f3fcc060360 2024-02-20T13:23:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.880+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fcc108040 con 0x7f3fcc060360 2024-02-20T13:23:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fc27fc700 1 -- 172.21.15.114:0/2930215150 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fbc01c320 con 0x7f3fcc060360 2024-02-20T13:23:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:28.883 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fb8031090 msgr2=0x7f3fb8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fb8031090 0x7f3fb8033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3fc80096a0 tx=0x7f3fc8006b40).stop 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 msgr2=0x7f3fcc106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc106750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f3fbc009f80 tx=0x7f3fbc00b040).stop 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 shutdown_connections 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3fb8031090 0x7f3fb8033550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 --2- 172.21.15.114:0/2930215150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fcc060360 0x7f3fcc106750 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:28.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 >> 172.21.15.114:0/2930215150 conn(0x7f3fcc0f0be0 msgr2=0x7f3fcc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:28.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 shutdown_connections 2024-02-20T13:23:28.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:28.882+0000 7f3fd4121700 1 -- 172.21.15.114:0/2930215150 wait complete. 2024-02-20T13:23:28.885 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:28.192050+0000 mgr.smithi038.powepi (mgr.14156) 188 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:28.279903+0000 mon.smithi038 (mon.0) 215 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:23:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:28.280150+0000 mon.smithi038 (mon.0) 216 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:23:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:28.882518+0000 mon.smithi038 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.114:0/2930215150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:30.629 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:30.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:31.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:30.192463+0000 mgr.smithi038.powepi (mgr.14156) 189 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:32.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 -- 172.21.15.114:0/2819034207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 msgr2=0x7f4c4c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 --2- 172.21.15.114:0/2819034207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c0f9190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f4c3c009fb0 tx=0x7f4c3c009b30).stop 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 -- 172.21.15.114:0/2819034207 shutdown_connections 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 --2- 172.21.15.114:0/2819034207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c0f9190 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 -- 172.21.15.114:0/2819034207 >> 172.21.15.114:0/2819034207 conn(0x7f4c4c0f0c10 msgr2=0x7f4c4c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 -- 172.21.15.114:0/2819034207 shutdown_connections 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.646+0000 7f4c532cc700 1 -- 172.21.15.114:0/2819034207 wait complete. 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c532cc700 1 Processor -- start 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c532cc700 1 -- start start 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c532cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c532cc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c106c60 con 0x7f4c4c060360 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c51068700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:32.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c51068700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34516/0 (socket says 172.21.15.114:34516) 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c51068700 1 -- 172.21.15.114:0/3705317809 learned_addr learned my addr 172.21.15.114:0/3705317809 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.647+0000 7f4c51068700 1 -- 172.21.15.114:0/3705317809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c005040 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.648+0000 7f4c51068700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c106770 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f4c3c009f80 tx=0x7f4c3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.648+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c3c009320 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.648+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4c3c013680 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.648+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c106e60 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.648+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c3c01eda0 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.648+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c1072f0 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.649+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4c3c01e420 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.649+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c045bb0 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.649+0000 7f4c41ffb700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4c38031090 0x7f4c38033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.649+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4c3c042350 con 0x7f4c4c060360 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.649+0000 7f4c50867700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4c38031090 0x7f4c38033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.650+0000 7f4c50867700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4c38031090 0x7f4c38033550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4c480096a0 tx=0x7f4c48006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:32.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.654+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4c3c01ccf0 con 0x7f4c4c060360 2024-02-20T13:23:32.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.863+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c4c1081a0 con 0x7f4c4c060360 2024-02-20T13:23:32.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.864+0000 7f4c41ffb700 1 -- 172.21.15.114:0/3705317809 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c3c01cb10 con 0x7f4c4c060360 2024-02-20T13:23:32.865 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:32.865 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:32.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4c38031090 msgr2=0x7f4c38033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4c38031090 0x7f4c38033550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4c480096a0 tx=0x7f4c48006b40).stop 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 msgr2=0x7f4c4c106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c106770 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f4c3c009f80 tx=0x7f4c3c00b040).stop 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 shutdown_connections 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4c38031090 0x7f4c38033550 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 --2- 172.21.15.114:0/3705317809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c060360 0x7f4c4c106770 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 >> 172.21.15.114:0/3705317809 conn(0x7f4c4c0f0c10 msgr2=0x7f4c4c0f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 shutdown_connections 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:32.865+0000 7f4c532cc700 1 -- 172.21.15.114:0/3705317809 wait complete. 2024-02-20T13:23:32.867 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:32.192875+0000 mgr.smithi038.powepi (mgr.14156) 190 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:33.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:32.864478+0000 mon.smithi038 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.114:0/3705317809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:34.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:34.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:34.193184+0000 mgr.smithi038.powepi (mgr.14156) 191 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 -- 172.21.15.114:0/2566446212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 msgr2=0x7f05ac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 --2- 172.21.15.114:0/2566446212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac0f69b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f05a0004760 tx=0x7f05a0009ad0).stop 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 -- 172.21.15.114:0/2566446212 shutdown_connections 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 --2- 172.21.15.114:0/2566446212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac0f69b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 -- 172.21.15.114:0/2566446212 >> 172.21.15.114:0/2566446212 conn(0x7f05ac0f1cd0 msgr2=0x7f05ac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 -- 172.21.15.114:0/2566446212 shutdown_connections 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 -- 172.21.15.114:0/2566446212 wait complete. 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 Processor -- start 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.802+0000 7f05b2c38700 1 -- start start 2024-02-20T13:23:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b2c38700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b2c38700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac106cb0 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b09d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b09d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:45768/0 (socket says 172.21.15.114:45768) 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b09d4700 1 -- 172.21.15.114:0/3446272570 learned_addr learned my addr 172.21.15.114:0/3446272570 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b09d4700 1 -- 172.21.15.114:0/3446272570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0009d70 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b09d4700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac1067c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f05a0004170 tx=0x7f05a0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a000e040 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac106eb0 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f05a0008ef0 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a0014c90 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.803+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac107340 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.804+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f05a00283f0 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.804+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac045bb0 con 0x7f05ac0f6590 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.805+0000 7f05a97fa700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f059c031090 0x7f059c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.805+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f05a00422f0 con 0x7f05ac0f6590 2024-02-20T13:23:36.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.805+0000 7f05abfff700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f059c031090 0x7f059c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:36.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.805+0000 7f05abfff700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f059c031090 0x7f059c033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f05980096a0 tx=0x7f0598006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:36.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:36.810+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f05a00266c0 con 0x7f05ac0f6590 2024-02-20T13:23:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.020+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05ac108190 con 0x7f05ac0f6590 2024-02-20T13:23:37.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.021+0000 7f05a97fa700 1 -- 172.21.15.114:0/3446272570 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f05a0026300 con 0x7f05ac0f6590 2024-02-20T13:23:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:37.021 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f059c031090 msgr2=0x7f059c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f059c031090 0x7f059c033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f05980096a0 tx=0x7f0598006b40).stop 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 msgr2=0x7f05ac1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac1067c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f05a0004170 tx=0x7f05a0009ad0).stop 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 shutdown_connections 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f059c031090 0x7f059c033550 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 --2- 172.21.15.114:0/3446272570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ac0f6590 0x7f05ac1067c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 >> 172.21.15.114:0/3446272570 conn(0x7f05ac0f1cd0 msgr2=0x7f05ac0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:37.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 shutdown_connections 2024-02-20T13:23:37.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:37.022+0000 7f05b2c38700 1 -- 172.21.15.114:0/3446272570 wait complete. 2024-02-20T13:23:37.024 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:36.193502+0000 mgr.smithi038.powepi (mgr.14156) 192 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:37.021413+0000 mon.smithi038 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.114:0/3446272570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:38.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:38.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:39.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:38.193914+0000 mgr.smithi038.powepi (mgr.14156) 193 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:40.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 -- 172.21.15.114:0/689886233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 msgr2=0x7f0a140f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 --2- 172.21.15.114:0/689886233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a140f69b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0a08004760 tx=0x7f0a08009ad0).stop 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 -- 172.21.15.114:0/689886233 shutdown_connections 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 --2- 172.21.15.114:0/689886233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a140f69b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 -- 172.21.15.114:0/689886233 >> 172.21.15.114:0/689886233 conn(0x7f0a140f1cd0 msgr2=0x7f0a140f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 -- 172.21.15.114:0/689886233 shutdown_connections 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.788+0000 7f0a1a643700 1 -- 172.21.15.114:0/689886233 wait complete. 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.789+0000 7f0a1a643700 1 Processor -- start 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.789+0000 7f0a1a643700 1 -- start start 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.789+0000 7f0a1a643700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a141067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.789+0000 7f0a1a643700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14106cb0 con 0x7f0a140f6590 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a13fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a141067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a13fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a141067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:45792/0 (socket says 172.21.15.114:45792) 2024-02-20T13:23:40.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a13fff700 1 -- 172.21.15.114:0/1782950093 learned_addr learned my addr 172.21.15.114:0/1782950093 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a13fff700 1 -- 172.21.15.114:0/1782950093 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a08009d70 con 0x7f0a140f6590 2024-02-20T13:23:40.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a13fff700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a141067c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f0a08004170 tx=0x7f0a08009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0800e040 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a14106eb0 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.790+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a14107340 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.791+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0a08008ef0 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.791+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a08014c90 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.791+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0a08014420 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.791+0000 7f0a10ff9700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09fc031090 0x7f09fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.791+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f4004fa0 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.792+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0a08042260 con 0x7f0a140f6590 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.792+0000 7f0a137fe700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09fc031090 0x7f09fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:40.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.792+0000 7f0a137fe700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09fc031090 0x7f09fc033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0a040096a0 tx=0x7f0a04006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:40.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:40.796+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0a08012d00 con 0x7f0a140f6590 2024-02-20T13:23:41.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.004+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f4005670 con 0x7f0a140f6590 2024-02-20T13:23:41.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.005+0000 7f0a10ff9700 1 -- 172.21.15.114:0/1782950093 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a08012b20 con 0x7f0a140f6590 2024-02-20T13:23:41.006 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:41.006 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09fc031090 msgr2=0x7f09fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09fc031090 0x7f09fc033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0a040096a0 tx=0x7f0a04006b40).stop 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 msgr2=0x7f0a141067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a141067c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f0a08004170 tx=0x7f0a08009ad0).stop 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 shutdown_connections 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f09fc031090 0x7f09fc033550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 --2- 172.21.15.114:0/1782950093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a140f6590 0x7f0a141067c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 >> 172.21.15.114:0/1782950093 conn(0x7f0a140f1cd0 msgr2=0x7f0a140f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 shutdown_connections 2024-02-20T13:23:41.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:41.007+0000 7f0a1a643700 1 -- 172.21.15.114:0/1782950093 wait complete. 2024-02-20T13:23:41.009 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:40.194292+0000 mgr.smithi038.powepi (mgr.14156) 194 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:41.005853+0000 mon.smithi038 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.114:0/1782950093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:42.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:42.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:43.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:42.194619+0000 mgr.smithi038.powepi (mgr.14156) 195 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:44.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 -- 172.21.15.114:0/4243725199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 msgr2=0x7fc3ec0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 --2- 172.21.15.114:0/4243725199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec0f69b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fc3dc004760 tx=0x7fc3dc009ad0).stop 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 -- 172.21.15.114:0/4243725199 shutdown_connections 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 --2- 172.21.15.114:0/4243725199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec0f69b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 -- 172.21.15.114:0/4243725199 >> 172.21.15.114:0/4243725199 conn(0x7fc3ec0f1cd0 msgr2=0x7fc3ec0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 -- 172.21.15.114:0/4243725199 shutdown_connections 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.784+0000 7fc3f32ac700 1 -- 172.21.15.114:0/4243725199 wait complete. 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f32ac700 1 Processor -- start 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f32ac700 1 -- start start 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f32ac700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f32ac700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec106cb0 con 0x7fc3ec0f6590 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f1048700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f1048700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:45796/0 (socket says 172.21.15.114:45796) 2024-02-20T13:23:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f1048700 1 -- 172.21.15.114:0/2607971193 learned_addr learned my addr 172.21.15.114:0/2607971193 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f1048700 1 -- 172.21.15.114:0/2607971193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc009d70 con 0x7fc3ec0f6590 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.785+0000 7fc3f1048700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec1067c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fc3dc013f10 tx=0x7fc3dc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.786+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3dc00e040 con 0x7fc3ec0f6590 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.786+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec106eb0 con 0x7fc3ec0f6590 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.786+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc3dc007df0 con 0x7fc3ec0f6590 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.786+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3dc014d60 con 0x7fc3ec0f6590 2024-02-20T13:23:44.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.786+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec107340 con 0x7fc3ec0f6590 2024-02-20T13:23:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.787+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc3dc014520 con 0x7fc3ec0f6590 2024-02-20T13:23:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.787+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec045bb0 con 0x7fc3ec0f6590 2024-02-20T13:23:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.787+0000 7fc3e1ffb700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc3d8031090 0x7fc3d8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.787+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc3dc042580 con 0x7fc3ec0f6590 2024-02-20T13:23:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.787+0000 7fc3f0847700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc3d8031090 0x7fc3d8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:44.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.787+0000 7fc3f0847700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc3d8031090 0x7fc3d8033550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc3e80096a0 tx=0x7fc3e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:44.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.792+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc3dc007960 con 0x7fc3ec0f6590 2024-02-20T13:23:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.998+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3ec1081f0 con 0x7fc3ec0f6590 2024-02-20T13:23:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:44.999+0000 7fc3e1ffb700 1 -- 172.21.15.114:0/2607971193 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3dc0262f0 con 0x7fc3ec0f6590 2024-02-20T13:23:45.000 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:45.000 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc3d8031090 msgr2=0x7fc3d8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc3d8031090 0x7fc3d8033550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc3e80096a0 tx=0x7fc3e8006b40).stop 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 msgr2=0x7fc3ec1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec1067c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fc3dc013f10 tx=0x7fc3dc009ad0).stop 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 shutdown_connections 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc3d8031090 0x7fc3d8033550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 --2- 172.21.15.114:0/2607971193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3ec0f6590 0x7fc3ec1067c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 >> 172.21.15.114:0/2607971193 conn(0x7fc3ec0f1cd0 msgr2=0x7fc3ec0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:45.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 shutdown_connections 2024-02-20T13:23:45.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:45.001+0000 7fc3f32ac700 1 -- 172.21.15.114:0/2607971193 wait complete. 2024-02-20T13:23:45.003 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:45.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:44.195011+0000 mgr.smithi038.powepi (mgr.14156) 196 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:45.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:44.999534+0000 mon.smithi038 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.114:0/2607971193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:46.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:46.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:47.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:46.195372+0000 mgr.smithi038.powepi (mgr.14156) 197 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 -- 172.21.15.114:0/900845525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 msgr2=0x7fa2540f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 --2- 172.21.15.114:0/900845525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f4d10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fa24c009960 tx=0x7fa24c004d10).stop 2024-02-20T13:23:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 -- 172.21.15.114:0/900845525 shutdown_connections 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 --2- 172.21.15.114:0/900845525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f4d10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 -- 172.21.15.114:0/900845525 >> 172.21.15.114:0/900845525 conn(0x7fa2540f0390 msgr2=0x7fa2540f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 -- 172.21.15.114:0/900845525 shutdown_connections 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 -- 172.21.15.114:0/900845525 wait complete. 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.832+0000 7fa259866700 1 Processor -- start 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa259866700 1 -- start start 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa259866700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa259866700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540f5960 con 0x7fa2540f48f0 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:36258/0 (socket says 172.21.15.114:36258) 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa252ffd700 1 -- 172.21.15.114:0/832262118 learned_addr learned my addr 172.21.15.114:0/832262118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa252ffd700 1 -- 172.21.15.114:0/832262118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c009640 con 0x7fa2540f48f0 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa252ffd700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f74e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fa24c009610 tx=0x7fa24c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:48.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.833+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24c018070 con 0x7fa2540f48f0 2024-02-20T13:23:48.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.834+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2540f5b00 con 0x7fa2540f48f0 2024-02-20T13:23:48.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.834+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa24c00b7e0 con 0x7fa2540f48f0 2024-02-20T13:23:48.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.834+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24c01d420 con 0x7fa2540f48f0 2024-02-20T13:23:48.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.834+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2540f5f90 con 0x7fa2540f48f0 2024-02-20T13:23:48.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.834+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa24c024060 con 0x7fa2540f48f0 2024-02-20T13:23:48.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.835+0000 7fa24b7fe700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa240031090 0x7fa240033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:48.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.835+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa24c0426e0 con 0x7fa2540f48f0 2024-02-20T13:23:48.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.835+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2540591e0 con 0x7fa2540f48f0 2024-02-20T13:23:48.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.835+0000 7fa2527fc700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa240031090 0x7fa240033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:48.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.835+0000 7fa2527fc700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa240031090 0x7fa240033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa23c0096a0 tx=0x7fa23c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:48.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:48.840+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa24c027440 con 0x7fa2540f48f0 2024-02-20T13:23:49.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.047+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2540fc3b0 con 0x7fa2540f48f0 2024-02-20T13:23:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.048+0000 7fa24b7fe700 1 -- 172.21.15.114:0/832262118 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa24c011320 con 0x7fa2540f48f0 2024-02-20T13:23:49.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:49.049 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa240031090 msgr2=0x7fa240033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa240031090 0x7fa240033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa23c0096a0 tx=0x7fa23c006b40).stop 2024-02-20T13:23:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 msgr2=0x7fa2540f74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f74e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fa24c009610 tx=0x7fa24c004d10).stop 2024-02-20T13:23:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 shutdown_connections 2024-02-20T13:23:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa240031090 0x7fa240033550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 --2- 172.21.15.114:0/832262118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2540f48f0 0x7fa2540f74e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 >> 172.21.15.114:0/832262118 conn(0x7fa2540f0390 msgr2=0x7fa2540f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 shutdown_connections 2024-02-20T13:23:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:49.050+0000 7fa259866700 1 -- 172.21.15.114:0/832262118 wait complete. 2024-02-20T13:23:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:48.195716+0000 mgr.smithi038.powepi (mgr.14156) 198 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:49.048908+0000 mon.smithi038 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.114:0/832262118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:50.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:50.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:51.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:50.196074+0000 mgr.smithi038.powepi (mgr.14156) 199 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 -- 172.21.15.114:0/159575657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 msgr2=0x7fa1ac0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 --2- 172.21.15.114:0/159575657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac0f56f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fa1a8009fb0 tx=0x7fa1a8009b30).stop 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 -- 172.21.15.114:0/159575657 shutdown_connections 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 --2- 172.21.15.114:0/159575657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac0f56f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 -- 172.21.15.114:0/159575657 >> 172.21.15.114:0/159575657 conn(0x7fa1ac0f0a30 msgr2=0x7fa1ac0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 -- 172.21.15.114:0/159575657 shutdown_connections 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.670+0000 7fa1b3583700 1 -- 172.21.15.114:0/159575657 wait complete. 2024-02-20T13:23:52.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b3583700 1 Processor -- start 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b3583700 1 -- start start 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b3583700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac10b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b3583700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac107120 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b2581700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac10b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b2581700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac10b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:36278/0 (socket says 172.21.15.114:36278) 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b2581700 1 -- 172.21.15.114:0/1930802387 learned_addr learned my addr 172.21.15.114:0/1930802387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b2581700 1 -- 172.21.15.114:0/1930802387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8005040 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.671+0000 7fa1b2581700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac10b730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fa1a80089f0 tx=0x7fa1a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.672+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a8009320 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.672+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa1a8013ec0 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.672+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac060300 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.672+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a8014d30 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.672+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac10bc20 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.672+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa1a80263a0 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.673+0000 7fa1a2ffd700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa198031090 0x7fa198033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.673+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa1a80424e0 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.673+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac0591e0 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.673+0000 7fa1b1d80700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa198031090 0x7fa198033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.673+0000 7fa1b1d80700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa198031090 0x7fa198033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa19c0096a0 tx=0x7fa19c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.678+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa1a80266d0 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.881+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1ac045bb0 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.882+0000 7fa1a2ffd700 1 -- 172.21.15.114:0/1930802387 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1a802b020 con 0x7fa1ac0f52d0 2024-02-20T13:23:52.884 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:52.884 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:52.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa198031090 msgr2=0x7fa198033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:52.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa198031090 0x7fa198033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa19c0096a0 tx=0x7fa19c006b40).stop 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 msgr2=0x7fa1ac10b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac10b730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fa1a80089f0 tx=0x7fa1a800b040).stop 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 shutdown_connections 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa198031090 0x7fa198033550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 --2- 172.21.15.114:0/1930802387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1ac0f52d0 0x7fa1ac10b730 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 >> 172.21.15.114:0/1930802387 conn(0x7fa1ac0f0a30 msgr2=0x7fa1ac0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 shutdown_connections 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:52.884+0000 7fa1b3583700 1 -- 172.21.15.114:0/1930802387 wait complete. 2024-02-20T13:23:52.886 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:53.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:52.196522+0000 mgr.smithi038.powepi (mgr.14156) 200 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:53.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:52.883873+0000 mon.smithi038 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.114:0/1930802387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:54.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:54.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:55.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:54.196805+0000 mgr.smithi038.powepi (mgr.14156) 201 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:56.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 -- 172.21.15.114:0/1721740902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea40f5480 msgr2=0x7feea40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:56.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 --2- 172.21.15.114:0/1721740902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea40f5480 0x7feea40f58a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fee98009fb0 tx=0x7fee98009b30).stop 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 -- 172.21.15.114:0/1721740902 shutdown_connections 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 --2- 172.21.15.114:0/1721740902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea40f5480 0x7feea40f58a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 -- 172.21.15.114:0/1721740902 >> 172.21.15.114:0/1721740902 conn(0x7feea40f0be0 msgr2=0x7feea40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 -- 172.21.15.114:0/1721740902 shutdown_connections 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 -- 172.21.15.114:0/1721740902 wait complete. 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.776+0000 7feeaa040700 1 Processor -- start 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feeaa040700 1 -- start start 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feeaa040700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 0x7feea4106ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:56.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feeaa040700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea41071d0 con 0x7feea41068c0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feea37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 0x7feea4106ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feea37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 0x7feea4106ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:51066/0 (socket says 172.21.15.114:51066) 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feea37fe700 1 -- 172.21.15.114:0/1032652917 learned_addr learned my addr 172.21.15.114:0/1032652917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.777+0000 7feea37fe700 1 -- 172.21.15.114:0/1032652917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee98005040 con 0x7feea41068c0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.778+0000 7feea37fe700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 0x7feea4106ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fee980134d0 tx=0x7fee98009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.778+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee98013ee0 con 0x7feea41068c0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.778+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea41073d0 con 0x7feea41068c0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.778+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fee98013680 con 0x7feea41068c0 2024-02-20T13:23:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.778+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee9801ee90 con 0x7feea41068c0 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.778+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea4107f20 con 0x7feea41068c0 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.779+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fee98025060 con 0x7feea41068c0 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.779+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea40591e0 con 0x7feea41068c0 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.779+0000 7fee8bfff700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee8c031040 0x7fee8c033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.779+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fee98042370 con 0x7feea41068c0 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.779+0000 7feea2ffd700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee8c031040 0x7fee8c033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:23:56.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.780+0000 7feea2ffd700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee8c031040 0x7fee8c033500 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fee940096a0 tx=0x7fee94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:23:56.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.784+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fee98028440 con 0x7feea41068c0 2024-02-20T13:23:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.996+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feea4045bb0 con 0x7feea41068c0 2024-02-20T13:23:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.997+0000 7fee8bfff700 1 -- 172.21.15.114:0/1032652917 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee9801c340 con 0x7feea41068c0 2024-02-20T13:23:56.999 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:23:56.999 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:23:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee8c031040 msgr2=0x7fee8c033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee8c031040 0x7fee8c033500 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fee940096a0 tx=0x7fee94006b40).stop 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 msgr2=0x7feea4106ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 0x7feea4106ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fee980134d0 tx=0x7fee98009b30).stop 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 shutdown_connections 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fee8c031040 0x7fee8c033500 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 --2- 172.21.15.114:0/1032652917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea41068c0 0x7feea4106ce0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 >> 172.21.15.114:0/1032652917 conn(0x7feea40f0be0 msgr2=0x7feea405ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 shutdown_connections 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:23:56.999+0000 7feeaa040700 1 -- 172.21.15.114:0/1032652917 wait complete. 2024-02-20T13:23:57.001 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:23:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:56.197137+0000 mgr.smithi038.powepi (mgr.14156) 202 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:23:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:23:56.998897+0000 mon.smithi038 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.114:0/1032652917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:23:58.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:23:58.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:23:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:23:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:23:58.197414+0000 mgr.smithi038.powepi (mgr.14156) 203 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 -- 172.21.15.114:0/3577579310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 msgr2=0x7f018c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 --2- 172.21.15.114:0/3577579310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c0f58a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f017c009fb0 tx=0x7f017c009b30).stop 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 -- 172.21.15.114:0/3577579310 shutdown_connections 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 --2- 172.21.15.114:0/3577579310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c0f58a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 -- 172.21.15.114:0/3577579310 >> 172.21.15.114:0/3577579310 conn(0x7f018c0f0be0 msgr2=0x7f018c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 -- 172.21.15.114:0/3577579310 shutdown_connections 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.941+0000 7f0193999700 1 -- 172.21.15.114:0/3577579310 wait complete. 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0193999700 1 Processor -- start 2024-02-20T13:24:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0193999700 1 -- start start 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0193999700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0193999700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c106cd0 con 0x7f018c0f5480 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0191735700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0191735700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:51082/0 (socket says 172.21.15.114:51082) 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0191735700 1 -- 172.21.15.114:0/1882816655 learned_addr learned my addr 172.21.15.114:0/1882816655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0191735700 1 -- 172.21.15.114:0/1882816655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c005040 con 0x7f018c0f5480 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.942+0000 7f0191735700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c1067e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f017c0089f0 tx=0x7f017c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.943+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f017c009320 con 0x7f018c0f5480 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.943+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f017c013ec0 con 0x7f018c0f5480 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.943+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c106ed0 con 0x7f018c0f5480 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.943+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f017c014d30 con 0x7f018c0f5480 2024-02-20T13:24:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.943+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c107360 con 0x7f018c0f5480 2024-02-20T13:24:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.944+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f017c0263a0 con 0x7f018c0f5480 2024-02-20T13:24:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.944+0000 7f0181ffb700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0178031090 0x7f0178033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.944+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f017c0424a0 con 0x7f018c0f5480 2024-02-20T13:24:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.944+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c0591e0 con 0x7f018c0f5480 2024-02-20T13:24:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.944+0000 7f0190f34700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0178031090 0x7f0178033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.944+0000 7f0190f34700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0178031090 0x7f0178033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f01880096a0 tx=0x7f0188006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:00.949+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f017c0266d0 con 0x7f018c0f5480 2024-02-20T13:24:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.162+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f018c045bb0 con 0x7f018c0f5480 2024-02-20T13:24:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.162+0000 7f0181ffb700 1 -- 172.21.15.114:0/1882816655 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f017c02b020 con 0x7f018c0f5480 2024-02-20T13:24:01.164 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:01.165 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0178031090 msgr2=0x7f0178033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0178031090 0x7f0178033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f01880096a0 tx=0x7f0188006b40).stop 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 msgr2=0x7f018c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c1067e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f017c0089f0 tx=0x7f017c00b040).stop 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 shutdown_connections 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0178031090 0x7f0178033550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 --2- 172.21.15.114:0/1882816655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f018c0f5480 0x7f018c1067e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 >> 172.21.15.114:0/1882816655 conn(0x7f018c0f0be0 msgr2=0x7f018c05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 shutdown_connections 2024-02-20T13:24:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:01.164+0000 7f0193999700 1 -- 172.21.15.114:0/1882816655 wait complete. 2024-02-20T13:24:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:01.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:00.197732+0000 mgr.smithi038.powepi (mgr.14156) 204 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:01.164018+0000 mon.smithi038 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.114:0/1882816655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:02.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:02.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:03.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:02.198092+0000 mgr.smithi038.powepi (mgr.14156) 205 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.723+0000 7f95743a6700 1 -- 172.21.15.114:0/430701137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 msgr2=0x7f956c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.723+0000 7f95743a6700 1 --2- 172.21.15.114:0/430701137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c0f69b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f955c004760 tx=0x7f955c009ad0).stop 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 -- 172.21.15.114:0/430701137 shutdown_connections 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 --2- 172.21.15.114:0/430701137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c0f69b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 -- 172.21.15.114:0/430701137 >> 172.21.15.114:0/430701137 conn(0x7f956c0f1cd0 msgr2=0x7f956c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 -- 172.21.15.114:0/430701137 shutdown_connections 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 -- 172.21.15.114:0/430701137 wait complete. 2024-02-20T13:24:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 Processor -- start 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 -- start start 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f95743a6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c106cb0 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f9572142700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f9572142700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:51102/0 (socket says 172.21.15.114:51102) 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.724+0000 7f9572142700 1 -- 172.21.15.114:0/3992215511 learned_addr learned my addr 172.21.15.114:0/3992215511 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f9572142700 1 -- 172.21.15.114:0/3992215511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c009d70 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f9572142700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c1067c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f955c004170 tx=0x7f955c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955c00e040 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c106eb0 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f955c008ef0 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955c014c90 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.725+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c107340 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.726+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f955c014420 con 0x7f956c0f6590 2024-02-20T13:24:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.726+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c045bb0 con 0x7f956c0f6590 2024-02-20T13:24:04.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.726+0000 7f9562ffd700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9558031090 0x7f9558033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:04.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.726+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f955c0421f0 con 0x7f956c0f6590 2024-02-20T13:24:04.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.726+0000 7f9571941700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9558031090 0x7f9558033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:04.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.727+0000 7f9571941700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9558031090 0x7f9558033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f95680096a0 tx=0x7f9568006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:04.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.731+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f955c026300 con 0x7f956c0f6590 2024-02-20T13:24:04.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.939+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f956c1081f0 con 0x7f956c0f6590 2024-02-20T13:24:04.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.939+0000 7f9562ffd700 1 -- 172.21.15.114:0/3992215511 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f955c012b20 con 0x7f956c0f6590 2024-02-20T13:24:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:04.941 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:04.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9558031090 msgr2=0x7f9558033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:04.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9558031090 0x7f9558033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f95680096a0 tx=0x7f9568006b40).stop 2024-02-20T13:24:04.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 msgr2=0x7f956c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c1067c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f955c004170 tx=0x7f955c009ad0).stop 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 shutdown_connections 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9558031090 0x7f9558033550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 --2- 172.21.15.114:0/3992215511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f956c0f6590 0x7f956c1067c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 >> 172.21.15.114:0/3992215511 conn(0x7f956c0f1cd0 msgr2=0x7f956c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 shutdown_connections 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:04.941+0000 7f95743a6700 1 -- 172.21.15.114:0/3992215511 wait complete. 2024-02-20T13:24:04.943 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:05.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:04.198507+0000 mgr.smithi038.powepi (mgr.14156) 206 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:05.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:04.940931+0000 mon.smithi038 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.114:0/3992215511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:06.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:06.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:07.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:06.198791+0000 mgr.smithi038.powepi (mgr.14156) 207 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:08.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 -- 172.21.15.114:0/1902518639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 msgr2=0x7fae640f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:08.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 --2- 172.21.15.114:0/1902518639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae640f9160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fae58009fb0 tx=0x7fae58009b30).stop 2024-02-20T13:24:08.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 -- 172.21.15.114:0/1902518639 shutdown_connections 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 --2- 172.21.15.114:0/1902518639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae640f9160 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 -- 172.21.15.114:0/1902518639 >> 172.21.15.114:0/1902518639 conn(0x7fae640f0be0 msgr2=0x7fae640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 -- 172.21.15.114:0/1902518639 shutdown_connections 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.710+0000 7fae69790700 1 -- 172.21.15.114:0/1902518639 wait complete. 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae69790700 1 Processor -- start 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae69790700 1 -- start start 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae69790700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae64106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae69790700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae64106c40 con 0x7fae64060360 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae62ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae64106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae62ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae64106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34706/0 (socket says 172.21.15.114:34706) 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae62ffd700 1 -- 172.21.15.114:0/4159321796 learned_addr learned my addr 172.21.15.114:0/4159321796 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae62ffd700 1 -- 172.21.15.114:0/4159321796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae58005040 con 0x7fae64060360 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.711+0000 7fae62ffd700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae64106750 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fae58009f80 tx=0x7fae5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.712+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae58009320 con 0x7fae64060360 2024-02-20T13:24:08.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.712+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae64106e40 con 0x7fae64060360 2024-02-20T13:24:08.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.712+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fae58013680 con 0x7fae64060360 2024-02-20T13:24:08.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.712+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae5801eda0 con 0x7fae64060360 2024-02-20T13:24:08.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.712+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641072d0 con 0x7fae64060360 2024-02-20T13:24:08.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.713+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fae5801e420 con 0x7fae64060360 2024-02-20T13:24:08.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.713+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae64045bb0 con 0x7fae64060360 2024-02-20T13:24:08.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.713+0000 7fae4b7fe700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fae4c031090 0x7fae4c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:08.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.713+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fae58042330 con 0x7fae64060360 2024-02-20T13:24:08.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.713+0000 7fae627fc700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fae4c031090 0x7fae4c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:08.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.713+0000 7fae627fc700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fae4c031090 0x7fae4c033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fae540096a0 tx=0x7fae54006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:08.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.718+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fae5801cb10 con 0x7fae64060360 2024-02-20T13:24:08.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.923+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae64108120 con 0x7fae64060360 2024-02-20T13:24:08.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.924+0000 7fae4b7fe700 1 -- 172.21.15.114:0/4159321796 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae5801cb10 con 0x7fae64060360 2024-02-20T13:24:08.926 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fae4c031090 msgr2=0x7fae4c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fae4c031090 0x7fae4c033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fae540096a0 tx=0x7fae54006b40).stop 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 msgr2=0x7fae64106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae64106750 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fae58009f80 tx=0x7fae5800b040).stop 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 shutdown_connections 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fae4c031090 0x7fae4c033550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:08.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 --2- 172.21.15.114:0/4159321796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae64060360 0x7fae64106750 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:08.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 >> 172.21.15.114:0/4159321796 conn(0x7fae640f0be0 msgr2=0x7fae640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:08.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 shutdown_connections 2024-02-20T13:24:08.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:08.926+0000 7fae69790700 1 -- 172.21.15.114:0/4159321796 wait complete. 2024-02-20T13:24:08.929 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:09.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:08.199050+0000 mgr.smithi038.powepi (mgr.14156) 208 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:08.925540+0000 mon.smithi038 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.114:0/4159321796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:10.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:10.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:11.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:10.199403+0000 mgr.smithi038.powepi (mgr.14156) 209 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 -- 172.21.15.114:0/1692438858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 msgr2=0x7f46680f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 --2- 172.21.15.114:0/1692438858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46680f4c70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4658009fb0 tx=0x7f4658009b30).stop 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 -- 172.21.15.114:0/1692438858 shutdown_connections 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 --2- 172.21.15.114:0/1692438858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46680f4c70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 -- 172.21.15.114:0/1692438858 >> 172.21.15.114:0/1692438858 conn(0x7f46680f0390 msgr2=0x7f46680f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 -- 172.21.15.114:0/1692438858 shutdown_connections 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.711+0000 7f466f5fd700 1 -- 172.21.15.114:0/1692438858 wait complete. 2024-02-20T13:24:12.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.712+0000 7f466f5fd700 1 Processor -- start 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.712+0000 7f466f5fd700 1 -- start start 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.712+0000 7f466f5fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46681067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.712+0000 7f466f5fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4668106ca0 con 0x7f46680f4850 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.712+0000 7f466d399700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46681067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.712+0000 7f466d399700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46681067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:34716/0 (socket says 172.21.15.114:34716) 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f466d399700 1 -- 172.21.15.114:0/2703307155 learned_addr learned my addr 172.21.15.114:0/2703307155 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f466d399700 1 -- 172.21.15.114:0/2703307155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658005040 con 0x7f46680f4850 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f466d399700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46681067b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f46580089f0 tx=0x7f465800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4658009320 con 0x7f46680f4850 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4668106ea0 con 0x7f46680f4850 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4658013ec0 con 0x7f46680f4850 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4658014d30 con 0x7f46680f4850 2024-02-20T13:24:12.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.713+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4668107330 con 0x7f46680f4850 2024-02-20T13:24:12.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.714+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f46580095f0 con 0x7f46680f4850 2024-02-20T13:24:12.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.714+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668045bb0 con 0x7f46680f4850 2024-02-20T13:24:12.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.714+0000 7f465dffb700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4654031090 0x7f4654033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:12.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.714+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f46580423e0 con 0x7f46680f4850 2024-02-20T13:24:12.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.715+0000 7f466cb98700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4654031090 0x7f4654033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:12.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.715+0000 7f466cb98700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4654031090 0x7f4654033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f46640096a0 tx=0x7f4664006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.720+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f465801d7f0 con 0x7f46680f4850 2024-02-20T13:24:12.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.924+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46681081e0 con 0x7f46680f4850 2024-02-20T13:24:12.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.925+0000 7f465dffb700 1 -- 172.21.15.114:0/2703307155 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f465801d7f0 con 0x7f46680f4850 2024-02-20T13:24:12.927 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:12.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4654031090 msgr2=0x7f4654033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4654031090 0x7f4654033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f46640096a0 tx=0x7f4664006b40).stop 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 msgr2=0x7f46681067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46681067b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f46580089f0 tx=0x7f465800b040).stop 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 shutdown_connections 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4654031090 0x7f4654033550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 --2- 172.21.15.114:0/2703307155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680f4850 0x7f46681067b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:12.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 >> 172.21.15.114:0/2703307155 conn(0x7f46680f0390 msgr2=0x7f46680f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:12.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 shutdown_connections 2024-02-20T13:24:12.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:12.927+0000 7f466f5fd700 1 -- 172.21.15.114:0/2703307155 wait complete. 2024-02-20T13:24:12.930 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:13.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:12.199718+0000 mgr.smithi038.powepi (mgr.14156) 210 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:13.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:12.926795+0000 mon.smithi038 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.114:0/2703307155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:14.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:14.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:15.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:14.200042+0000 mgr.smithi038.powepi (mgr.14156) 211 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 -- 172.21.15.114:0/592999509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 msgr2=0x7fad980f4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 --2- 172.21.15.114:0/592999509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad980f4540 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fad94009fb0 tx=0x7fad94009b30).stop 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 -- 172.21.15.114:0/592999509 shutdown_connections 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 --2- 172.21.15.114:0/592999509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad980f4540 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 -- 172.21.15.114:0/592999509 >> 172.21.15.114:0/592999509 conn(0x7fad980efd80 msgr2=0x7fad980f21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 -- 172.21.15.114:0/592999509 shutdown_connections 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.612+0000 7fad9e8b8700 1 -- 172.21.15.114:0/592999509 wait complete. 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9e8b8700 1 Processor -- start 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9e8b8700 1 -- start start 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9e8b8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad981065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9e8b8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad98106ac0 con 0x7fad980f4120 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9d8b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad981065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9d8b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad981065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55686/0 (socket says 172.21.15.114:55686) 2024-02-20T13:24:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9d8b6700 1 -- 172.21.15.114:0/652483249 learned_addr learned my addr 172.21.15.114:0/652483249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9d8b6700 1 -- 172.21.15.114:0/652483249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94005040 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.613+0000 7fad9d8b6700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad981065d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fad94009f80 tx=0x7fad9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.614+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad94009320 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.614+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad98106cc0 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.614+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad94013680 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.614+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad9401eda0 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.614+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad98107150 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.614+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fad94014420 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.615+0000 7fad8e7fc700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad84031090 0x7fad84033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.615+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fad94042370 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.615+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad980591e0 con 0x7fad980f4120 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.615+0000 7fad9d0b5700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad84031090 0x7fad84033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.615+0000 7fad9d0b5700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad84031090 0x7fad84033550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fad880096a0 tx=0x7fad88006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.620+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad9401c910 con 0x7fad980f4120 2024-02-20T13:24:16.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.828+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad980fbe30 con 0x7fad980f4120 2024-02-20T13:24:16.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.829+0000 7fad8e7fc700 1 -- 172.21.15.114:0/652483249 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad94026350 con 0x7fad980f4120 2024-02-20T13:24:16.830 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:16.830 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:16.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.830+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad84031090 msgr2=0x7fad84033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:16.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad84031090 0x7fad84033550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fad880096a0 tx=0x7fad88006b40).stop 2024-02-20T13:24:16.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 msgr2=0x7fad981065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:16.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad981065d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fad94009f80 tx=0x7fad9400b040).stop 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 shutdown_connections 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad84031090 0x7fad84033550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 --2- 172.21.15.114:0/652483249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad980f4120 0x7fad981065d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 >> 172.21.15.114:0/652483249 conn(0x7fad980efd80 msgr2=0x7fad980f0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 shutdown_connections 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:16.831+0000 7fad9e8b8700 1 -- 172.21.15.114:0/652483249 wait complete. 2024-02-20T13:24:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:17.674 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:16.200429+0000 mgr.smithi038.powepi (mgr.14156) 212 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:17.674 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:16.829865+0000 mon.smithi038 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.114:0/652483249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:18.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:18.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:19.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:18.200803+0000 mgr.smithi038.powepi (mgr.14156) 213 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:20.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2056694597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 msgr2=0x7fbbc40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:20.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 --2- 172.21.15.114:0/2056694597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc40f56f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fbbb4009fb0 tx=0x7fbbb4009b30).stop 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2056694597 shutdown_connections 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 --2- 172.21.15.114:0/2056694597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc40f56f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2056694597 >> 172.21.15.114:0/2056694597 conn(0x7fbbc40f0a30 msgr2=0x7fbbc40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2056694597 shutdown_connections 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2056694597 wait complete. 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 Processor -- start 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- start start 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc4106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.759+0000 7fbbc8a9c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4106b30 con 0x7fbbc40f52d0 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc4106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc4106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55710/0 (socket says 172.21.15.114:55710) 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc37fe700 1 -- 172.21.15.114:0/2339333592 learned_addr learned my addr 172.21.15.114:0/2339333592 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc37fe700 1 -- 172.21.15.114:0/2339333592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4005040 con 0x7fbbc40f52d0 2024-02-20T13:24:20.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc37fe700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc4106640 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbbb40089f0 tx=0x7fbbb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb4009320 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc4106d30 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbbb4013ec0 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb4014d30 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.760+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41071c0 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.761+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbbb40095f0 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.761+0000 7fbbabfff700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbac031090 0x7fbbac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.761+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc4045bb0 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.761+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbbb40424e0 con 0x7fbbc40f52d0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.762+0000 7fbbc2ffd700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbac031090 0x7fbbac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:20.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.762+0000 7fbbc2ffd700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbac031090 0x7fbbac033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbbb80096a0 tx=0x7fbbb8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.766+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbbb400fe40 con 0x7fbbc40f52d0 2024-02-20T13:24:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.974+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbc4108010 con 0x7fbbc40f52d0 2024-02-20T13:24:20.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.975+0000 7fbbabfff700 1 -- 172.21.15.114:0/2339333592 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbbb400fe40 con 0x7fbbc40f52d0 2024-02-20T13:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:20.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbac031090 msgr2=0x7fbbac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:20.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbac031090 0x7fbbac033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbbb80096a0 tx=0x7fbbb8006b40).stop 2024-02-20T13:24:20.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 msgr2=0x7fbbc4106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:20.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc4106640 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbbb40089f0 tx=0x7fbbb400b040).stop 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 shutdown_connections 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbbac031090 0x7fbbac033550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 --2- 172.21.15.114:0/2339333592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbc40f52d0 0x7fbbc4106640 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.976+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 >> 172.21.15.114:0/2339333592 conn(0x7fbbc40f0a30 msgr2=0x7fbbc40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.977+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 shutdown_connections 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:20.977+0000 7fbbc8a9c700 1 -- 172.21.15.114:0/2339333592 wait complete. 2024-02-20T13:24:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:21.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:20.201182+0000 mgr.smithi038.powepi (mgr.14156) 214 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:21.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:20.975322+0000 mon.smithi038 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.114:0/2339333592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:22.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:22.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:22.201507+0000 mgr.smithi038.powepi (mgr.14156) 215 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 -- 172.21.15.114:0/404568636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 msgr2=0x7f0cd80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 --2- 172.21.15.114:0/404568636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd80f69b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f0ccc004760 tx=0x7f0ccc009ad0).stop 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 -- 172.21.15.114:0/404568636 shutdown_connections 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 --2- 172.21.15.114:0/404568636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd80f69b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 -- 172.21.15.114:0/404568636 >> 172.21.15.114:0/404568636 conn(0x7f0cd80f1cd0 msgr2=0x7f0cd80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 -- 172.21.15.114:0/404568636 shutdown_connections 2024-02-20T13:24:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.658+0000 7f0cdd112700 1 -- 172.21.15.114:0/404568636 wait complete. 2024-02-20T13:24:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cdd112700 1 Processor -- start 2024-02-20T13:24:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cdd112700 1 -- start start 2024-02-20T13:24:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cdd112700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd8108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cdd112700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8108f00 con 0x7f0cd80f6590 2024-02-20T13:24:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cd6d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd8108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cd6d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd8108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55724/0 (socket says 172.21.15.114:55724) 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cd6d9d700 1 -- 172.21.15.114:0/2241059128 learned_addr learned my addr 172.21.15.114:0/2241059128 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cd6d9d700 1 -- 172.21.15.114:0/2241059128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc009d70 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cd6d9d700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd8108a10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0ccc004170 tx=0x7f0ccc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.659+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ccc00e040 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.660+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd8109100 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.660+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0ccc008ef0 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.660+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ccc014c90 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.660+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd8109590 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.660+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0ccc014420 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.660+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8045bb0 con 0x7f0cd80f6590 2024-02-20T13:24:24.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.661+0000 7f0cbf7fe700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cc0031090 0x7f0cc0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:24.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.661+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0ccc042320 con 0x7f0cd80f6590 2024-02-20T13:24:24.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.661+0000 7f0cd659c700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cc0031090 0x7f0cc0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:24.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.661+0000 7f0cd659c700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cc0031090 0x7f0cc0033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0cc80096a0 tx=0x7f0cc8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.666+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0ccc012d00 con 0x7f0cd80f6590 2024-02-20T13:24:24.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.871+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cd810a390 con 0x7f0cd80f6590 2024-02-20T13:24:24.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.872+0000 7f0cbf7fe700 1 -- 172.21.15.114:0/2241059128 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ccc012b20 con 0x7f0cd80f6590 2024-02-20T13:24:24.872 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:24.873 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cc0031090 msgr2=0x7f0cc0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cc0031090 0x7f0cc0033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0cc80096a0 tx=0x7f0cc8006b40).stop 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 msgr2=0x7f0cd8108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd8108a10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0ccc004170 tx=0x7f0ccc009ad0).stop 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 shutdown_connections 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0cc0031090 0x7f0cc0033550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 --2- 172.21.15.114:0/2241059128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cd80f6590 0x7f0cd8108a10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 >> 172.21.15.114:0/2241059128 conn(0x7f0cd80f1cd0 msgr2=0x7f0cd80f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 shutdown_connections 2024-02-20T13:24:24.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:24.873+0000 7f0cdd112700 1 -- 172.21.15.114:0/2241059128 wait complete. 2024-02-20T13:24:24.875 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:24.201838+0000 mgr.smithi038.powepi (mgr.14156) 216 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:24.872227+0000 mon.smithi038 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.114:0/2241059128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:26.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:26.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:27.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:26.202249+0000 mgr.smithi038.powepi (mgr.14156) 217 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:28.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 -- 172.21.15.114:0/3502481460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 msgr2=0x7f82700f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 --2- 172.21.15.114:0/3502481460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82700f69b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8264004760 tx=0x7f8264009ad0).stop 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 -- 172.21.15.114:0/3502481460 shutdown_connections 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 --2- 172.21.15.114:0/3502481460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82700f69b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 -- 172.21.15.114:0/3502481460 >> 172.21.15.114:0/3502481460 conn(0x7f82700f1cd0 msgr2=0x7f82700f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 -- 172.21.15.114:0/3502481460 shutdown_connections 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 -- 172.21.15.114:0/3502481460 wait complete. 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.587+0000 7f8274a1c700 1 Processor -- start 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f8274a1c700 1 -- start start 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f8274a1c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82701067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f8274a1c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270106cb0 con 0x7f82700f6590 2024-02-20T13:24:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f826e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82701067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f826e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82701067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:56314/0 (socket says 172.21.15.114:56314) 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f826e59c700 1 -- 172.21.15.114:0/4291029924 learned_addr learned my addr 172.21.15.114:0/4291029924 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f826e59c700 1 -- 172.21.15.114:0/4291029924 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264009d70 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f826e59c700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82701067c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f8264004170 tx=0x7f8264009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f826400e040 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.588+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8270106eb0 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.589+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8264008ef0 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.589+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8270107340 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.589+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8264014c90 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.589+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8264014420 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.589+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270045bb0 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.589+0000 7f825effd700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f82580310e0 0x7f82580335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.590+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8264042280 con 0x7f82700f6590 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.590+0000 7f826dd9b700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f82580310e0 0x7f82580335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.590+0000 7f826dd9b700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f82580310e0 0x7f82580335a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f82600096a0 tx=0x7f8260006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:28.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.594+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8264026300 con 0x7f82700f6590 2024-02-20T13:24:28.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.802+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8270108190 con 0x7f82700f6590 2024-02-20T13:24:28.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.803+0000 7f825effd700 1 -- 172.21.15.114:0/4291029924 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8264012b20 con 0x7f82700f6590 2024-02-20T13:24:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:28.804 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f82580310e0 msgr2=0x7f82580335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f82580310e0 0x7f82580335a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f82600096a0 tx=0x7f8260006b40).stop 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 msgr2=0x7f82701067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82701067c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f8264004170 tx=0x7f8264009ad0).stop 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 shutdown_connections 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f82580310e0 0x7f82580335a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 --2- 172.21.15.114:0/4291029924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700f6590 0x7f82701067c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 >> 172.21.15.114:0/4291029924 conn(0x7f82700f1cd0 msgr2=0x7f82700f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 shutdown_connections 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:28.805+0000 7f8274a1c700 1 -- 172.21.15.114:0/4291029924 wait complete. 2024-02-20T13:24:28.806 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:28.202730+0000 mgr.smithi038.powepi (mgr.14156) 218 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:28.281220+0000 mon.smithi038 (mon.0) 232 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:24:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:28.281480+0000 mon.smithi038 (mon.0) 233 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:24:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:28.803720+0000 mon.smithi038 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.114:0/4291029924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:30.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:30.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:31.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:30.203137+0000 mgr.smithi038.powepi (mgr.14156) 219 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 -- 172.21.15.114:0/2263708920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 msgr2=0x7fe7140f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 --2- 172.21.15.114:0/2263708920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe7140f69b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fe708009fb0 tx=0x7fe708009b30).stop 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 -- 172.21.15.114:0/2263708920 shutdown_connections 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 --2- 172.21.15.114:0/2263708920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe7140f69b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 -- 172.21.15.114:0/2263708920 >> 172.21.15.114:0/2263708920 conn(0x7fe7140f1cf0 msgr2=0x7fe7140f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 -- 172.21.15.114:0/2263708920 shutdown_connections 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 -- 172.21.15.114:0/2263708920 wait complete. 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.711+0000 7fe71ad2f700 1 Processor -- start 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe71ad2f700 1 -- start start 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe71ad2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe71406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:32.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe71ad2f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe714069f00 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe718acb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe71406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe718acb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe71406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:56320/0 (socket says 172.21.15.114:56320) 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe718acb700 1 -- 172.21.15.114:0/1161170517 learned_addr learned my addr 172.21.15.114:0/1161170517 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.712+0000 7fe718acb700 1 -- 172.21.15.114:0/1161170517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe708005040 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.713+0000 7fe718acb700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe71406ba80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fe7080089f0 tx=0x7fe70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.713+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe708009320 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.713+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71406a0a0 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.713+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe708013ec0 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.713+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe708014d30 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.713+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71406a530 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.714+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe7080095f0 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.714+0000 7fe7117fa700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe704031090 0x7fe704033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.714+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7140591e0 con 0x7fe7140f6590 2024-02-20T13:24:32.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.714+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe7080423e0 con 0x7fe7140f6590 2024-02-20T13:24:32.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.714+0000 7fe713fff700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe704031090 0x7fe704033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:32.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.714+0000 7fe713fff700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe704031090 0x7fe704033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe7000096a0 tx=0x7fe700006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:32.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.719+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe70801d7f0 con 0x7fe7140f6590 2024-02-20T13:24:32.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.928+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7140fc080 con 0x7fe7140f6590 2024-02-20T13:24:32.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.929+0000 7fe7117fa700 1 -- 172.21.15.114:0/1161170517 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe70801d7f0 con 0x7fe7140f6590 2024-02-20T13:24:32.929 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:32.929 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe704031090 msgr2=0x7fe704033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe704031090 0x7fe704033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe7000096a0 tx=0x7fe700006b40).stop 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 msgr2=0x7fe71406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe71406ba80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fe7080089f0 tx=0x7fe70800b040).stop 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 shutdown_connections 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe704031090 0x7fe704033550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 --2- 172.21.15.114:0/1161170517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7140f6590 0x7fe71406ba80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 >> 172.21.15.114:0/1161170517 conn(0x7fe7140f1cf0 msgr2=0x7fe7140f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:32.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 shutdown_connections 2024-02-20T13:24:32.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:32.930+0000 7fe71ad2f700 1 -- 172.21.15.114:0/1161170517 wait complete. 2024-02-20T13:24:32.932 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:33.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:32.203573+0000 mgr.smithi038.powepi (mgr.14156) 220 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:33.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:32.929222+0000 mon.smithi038 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.114:0/1161170517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:34.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:34.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:34.203963+0000 mgr.smithi038.powepi (mgr.14156) 221 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:36.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 -- 172.21.15.114:0/4143124950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 msgr2=0x7fb1940f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 --2- 172.21.15.114:0/4143124950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb1940f69b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fb184004760 tx=0x7fb184009ad0).stop 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 -- 172.21.15.114:0/4143124950 shutdown_connections 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 --2- 172.21.15.114:0/4143124950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb1940f69b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 -- 172.21.15.114:0/4143124950 >> 172.21.15.114:0/4143124950 conn(0x7fb1940f1cd0 msgr2=0x7fb1940f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 -- 172.21.15.114:0/4143124950 shutdown_connections 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.739+0000 7fb19c456700 1 -- 172.21.15.114:0/4143124950 wait complete. 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19c456700 1 Processor -- start 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19c456700 1 -- start start 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19c456700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb19406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19c456700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb194069f00 con 0x7fb1940f6590 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19a1f2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb19406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19a1f2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb19406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54932/0 (socket says 172.21.15.114:54932) 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.740+0000 7fb19a1f2700 1 -- 172.21.15.114:0/1394459445 learned_addr learned my addr 172.21.15.114:0/1394459445 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb19a1f2700 1 -- 172.21.15.114:0/1394459445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb184009d70 con 0x7fb1940f6590 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb19a1f2700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb19406ba80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb184004170 tx=0x7fb184009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb18400e040 con 0x7fb1940f6590 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19406a0a0 con 0x7fb1940f6590 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb184008ef0 con 0x7fb1940f6590 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb184014c90 con 0x7fb1940f6590 2024-02-20T13:24:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.741+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19406a530 con 0x7fb1940f6590 2024-02-20T13:24:36.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.742+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb184014420 con 0x7fb1940f6590 2024-02-20T13:24:36.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.742+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1940591e0 con 0x7fb1940f6590 2024-02-20T13:24:36.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.742+0000 7fb18affd700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb180031090 0x7fb180033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:36.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.742+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb184042210 con 0x7fb1940f6590 2024-02-20T13:24:36.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.742+0000 7fb1999f1700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb180031090 0x7fb180033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:36.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.743+0000 7fb1999f1700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb180031090 0x7fb180033550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb1900096a0 tx=0x7fb190006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:36.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.747+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb184026300 con 0x7fb1940f6590 2024-02-20T13:24:36.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.957+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb194045bb0 con 0x7fb1940f6590 2024-02-20T13:24:36.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.958+0000 7fb18affd700 1 -- 172.21.15.114:0/1394459445 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb184012b20 con 0x7fb1940f6590 2024-02-20T13:24:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:36.959 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:36.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb180031090 msgr2=0x7fb180033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:36.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb180031090 0x7fb180033550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb1900096a0 tx=0x7fb190006b40).stop 2024-02-20T13:24:36.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 msgr2=0x7fb19406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb19406ba80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb184004170 tx=0x7fb184009ad0).stop 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 shutdown_connections 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb180031090 0x7fb180033550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 --2- 172.21.15.114:0/1394459445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1940f6590 0x7fb19406ba80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 >> 172.21.15.114:0/1394459445 conn(0x7fb1940f1cd0 msgr2=0x7fb1940f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 shutdown_connections 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:36.960+0000 7fb19c456700 1 -- 172.21.15.114:0/1394459445 wait complete. 2024-02-20T13:24:36.961 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:37.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:36.204390+0000 mgr.smithi038.powepi (mgr.14156) 222 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:36.958681+0000 mon.smithi038 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.114:0/1394459445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:38.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:38.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:39.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:38.204879+0000 mgr.smithi038.powepi (mgr.14156) 223 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 -- 172.21.15.114:0/3269358036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 msgr2=0x7f9fa80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 --2- 172.21.15.114:0/3269358036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa80f58a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f9f9c009fb0 tx=0x7f9f9c009b30).stop 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 -- 172.21.15.114:0/3269358036 shutdown_connections 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 --2- 172.21.15.114:0/3269358036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa80f58a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 -- 172.21.15.114:0/3269358036 >> 172.21.15.114:0/3269358036 conn(0x7f9fa80f0be0 msgr2=0x7f9fa80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 -- 172.21.15.114:0/3269358036 shutdown_connections 2024-02-20T13:24:40.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.446+0000 7f9faee2c700 1 -- 172.21.15.114:0/3269358036 wait complete. 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9faee2c700 1 Processor -- start 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9faee2c700 1 -- start start 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9faee2c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9faee2c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8106cd0 con 0x7f9fa80f5480 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9facbc8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9facbc8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54954/0 (socket says 172.21.15.114:54954) 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.447+0000 7f9facbc8700 1 -- 172.21.15.114:0/4023232849 learned_addr learned my addr 172.21.15.114:0/4023232849 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9facbc8700 1 -- 172.21.15.114:0/4023232849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c005040 con 0x7f9fa80f5480 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9facbc8700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa81067e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9f9c0089f0 tx=0x7f9f9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f9c009320 con 0x7f9fa80f5480 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa8106ed0 con 0x7f9fa80f5480 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9f9c013ec0 con 0x7f9fa80f5480 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f9c014d30 con 0x7f9fa80f5480 2024-02-20T13:24:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.448+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa8107360 con 0x7f9fa80f5480 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.449+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9f9c014420 con 0x7f9fa80f5480 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.449+0000 7f9fa57fa700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9f98031090 0x7f9f98033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.449+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8045bb0 con 0x7f9fa80f5480 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.449+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9f9c0424c0 con 0x7f9fa80f5480 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.449+0000 7f9fa7fff700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9f98031090 0x7f9f98033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.450+0000 7f9fa7fff700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9f98031090 0x7f9f98033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9f940096a0 tx=0x7f9f94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:40.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.454+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9f9c01d980 con 0x7f9fa80f5480 2024-02-20T13:24:40.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.660+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fa81081e0 con 0x7f9fa80f5480 2024-02-20T13:24:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.661+0000 7f9fa57fa700 1 -- 172.21.15.114:0/4023232849 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f9c02c090 con 0x7f9fa80f5480 2024-02-20T13:24:40.663 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:40.663 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9f98031090 msgr2=0x7f9f98033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9f98031090 0x7f9f98033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9f940096a0 tx=0x7f9f94006b40).stop 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 msgr2=0x7f9fa81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa81067e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9f9c0089f0 tx=0x7f9f9c00b040).stop 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 shutdown_connections 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9f98031090 0x7f9f98033550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 --2- 172.21.15.114:0/4023232849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fa80f5480 0x7f9fa81067e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 >> 172.21.15.114:0/4023232849 conn(0x7f9fa80f0be0 msgr2=0x7f9fa805ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 shutdown_connections 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:40.663+0000 7f9faee2c700 1 -- 172.21.15.114:0/4023232849 wait complete. 2024-02-20T13:24:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:40.205363+0000 mgr.smithi038.powepi (mgr.14156) 224 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:40.662715+0000 mon.smithi038 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.114:0/4023232849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:42.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:42.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:43.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:42.205616+0000 mgr.smithi038.powepi (mgr.14156) 225 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:44.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 -- 172.21.15.114:0/311156199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 msgr2=0x7f00840f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 --2- 172.21.15.114:0/311156199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f00840f6f20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0080009fb0 tx=0x7f0080009b30).stop 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 -- 172.21.15.114:0/311156199 shutdown_connections 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 --2- 172.21.15.114:0/311156199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f00840f6f20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 -- 172.21.15.114:0/311156199 >> 172.21.15.114:0/311156199 conn(0x7f00840f0360 msgr2=0x7f00840f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 -- 172.21.15.114:0/311156199 shutdown_connections 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 -- 172.21.15.114:0/311156199 wait complete. 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 Processor -- start 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.465+0000 7f008c63b700 1 -- start start 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008c63b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f0084109490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008c63b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00840f53b0 con 0x7f00840f6b00 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008a3d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f0084109490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:44.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008a3d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f0084109490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54972/0 (socket says 172.21.15.114:54972) 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008a3d7700 1 -- 172.21.15.114:0/26057438 learned_addr learned my addr 172.21.15.114:0/26057438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008a3d7700 1 -- 172.21.15.114:0/26057438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080005040 con 0x7f00840f6b00 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.466+0000 7f008a3d7700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f0084109490 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f0080009f80 tx=0x7f008000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.467+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0080009320 con 0x7f00840f6b00 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.467+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00840f55b0 con 0x7f00840f6b00 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.467+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0080013680 con 0x7f00840f6b00 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.467+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f008001eda0 con 0x7f00840f6b00 2024-02-20T13:24:44.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.467+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0084109b40 con 0x7f00840f6b00 2024-02-20T13:24:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.467+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f008001e420 con 0x7f00840f6b00 2024-02-20T13:24:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.468+0000 7f007affd700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0070031090 0x7f0070033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.468+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0084045bb0 con 0x7f00840f6b00 2024-02-20T13:24:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.468+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0080042370 con 0x7f00840f6b00 2024-02-20T13:24:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.468+0000 7f0089bd6700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0070031090 0x7f0070033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.468+0000 7f0089bd6700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0070031090 0x7f0070033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f00740096a0 tx=0x7f0074006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:44.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.473+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f008001ccf0 con 0x7f00840f6b00 2024-02-20T13:24:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.680+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f008410a360 con 0x7f00840f6b00 2024-02-20T13:24:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.680+0000 7f007affd700 1 -- 172.21.15.114:0/26057438 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f008001cb10 con 0x7f00840f6b00 2024-02-20T13:24:44.682 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:44.683 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0070031090 msgr2=0x7f0070033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0070031090 0x7f0070033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f00740096a0 tx=0x7f0074006b40).stop 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 msgr2=0x7f0084109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f0084109490 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f0080009f80 tx=0x7f008000b040).stop 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 shutdown_connections 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0070031090 0x7f0070033550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 --2- 172.21.15.114:0/26057438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00840f6b00 0x7f0084109490 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 >> 172.21.15.114:0/26057438 conn(0x7f00840f0360 msgr2=0x7f00840f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 shutdown_connections 2024-02-20T13:24:44.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:44.682+0000 7f008c63b700 1 -- 172.21.15.114:0/26057438 wait complete. 2024-02-20T13:24:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:45.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:44.206003+0000 mgr.smithi038.powepi (mgr.14156) 226 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:45.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:44.682196+0000 mon.smithi038 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.114:0/26057438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:46.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:46.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:47.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:46.206453+0000 mgr.smithi038.powepi (mgr.14156) 227 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 -- 172.21.15.114:0/254566444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 msgr2=0x7f4d000f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 --2- 172.21.15.114:0/254566444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d000f58a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f4cf0009fb0 tx=0x7f4cf0009b30).stop 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 -- 172.21.15.114:0/254566444 shutdown_connections 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 --2- 172.21.15.114:0/254566444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d000f58a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 -- 172.21.15.114:0/254566444 >> 172.21.15.114:0/254566444 conn(0x7f4d000f0be0 msgr2=0x7f4d000f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 -- 172.21.15.114:0/254566444 shutdown_connections 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.301+0000 7f4d07a11700 1 -- 172.21.15.114:0/254566444 wait complete. 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d07a11700 1 Processor -- start 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d07a11700 1 -- start start 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d07a11700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d001067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d07a11700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d00106cb0 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d057ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d001067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d057ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d001067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44608/0 (socket says 172.21.15.114:44608) 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d057ad700 1 -- 172.21.15.114:0/3445818020 learned_addr learned my addr 172.21.15.114:0/3445818020 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.302+0000 7f4d057ad700 1 -- 172.21.15.114:0/3445818020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf0005040 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.303+0000 7f4d057ad700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d001067c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4cf00089f0 tx=0x7f4cf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.303+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cf0009320 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.303+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d00106eb0 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.303+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4cf0013ec0 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.303+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cf0014d30 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.303+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d00107340 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.304+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4cf00095f0 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.304+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d00045bb0 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.304+0000 7f4cf5ffb700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4cec031090 0x7f4cec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.304+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4cf00424e0 con 0x7f4d000f5480 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.304+0000 7f4d04fac700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4cec031090 0x7f4cec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.305+0000 7f4d04fac700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4cec031090 0x7f4cec033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4cfc0096a0 tx=0x7f4cfc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.309+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4cf001d7f0 con 0x7f4d000f5480 2024-02-20T13:24:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.517+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d001081f0 con 0x7f4d000f5480 2024-02-20T13:24:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.518+0000 7f4cf5ffb700 1 -- 172.21.15.114:0/3445818020 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4cf000fe40 con 0x7f4d000f5480 2024-02-20T13:24:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:48.520 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4cec031090 msgr2=0x7f4cec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4cec031090 0x7f4cec033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4cfc0096a0 tx=0x7f4cfc006b40).stop 2024-02-20T13:24:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 msgr2=0x7f4d001067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d001067c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4cf00089f0 tx=0x7f4cf000b040).stop 2024-02-20T13:24:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 shutdown_connections 2024-02-20T13:24:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4cec031090 0x7f4cec033550 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 --2- 172.21.15.114:0/3445818020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d000f5480 0x7f4d001067c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 >> 172.21.15.114:0/3445818020 conn(0x7f4d000f0be0 msgr2=0x7f4d0005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.519+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 shutdown_connections 2024-02-20T13:24:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:48.520+0000 7f4d07a11700 1 -- 172.21.15.114:0/3445818020 wait complete. 2024-02-20T13:24:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:48.206828+0000 mgr.smithi038.powepi (mgr.14156) 228 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:48.519590+0000 mon.smithi038 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.114:0/3445818020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:50.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:50.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:51.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:50.207257+0000 mgr.smithi038.powepi (mgr.14156) 229 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.162+0000 7f396359e700 1 -- 172.21.15.114:0/3627678892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 msgr2=0x7f39640f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.162+0000 7f396359e700 1 --2- 172.21.15.114:0/3627678892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f39640f56f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f394c009fb0 tx=0x7f394c009b30).stop 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.162+0000 7f396359e700 1 -- 172.21.15.114:0/3627678892 shutdown_connections 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.162+0000 7f396359e700 1 --2- 172.21.15.114:0/3627678892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f39640f56f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.162+0000 7f396359e700 1 -- 172.21.15.114:0/3627678892 >> 172.21.15.114:0/3627678892 conn(0x7f39640f0a30 msgr2=0x7f39640f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396359e700 1 -- 172.21.15.114:0/3627678892 shutdown_connections 2024-02-20T13:24:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396359e700 1 -- 172.21.15.114:0/3627678892 wait complete. 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396359e700 1 Processor -- start 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396359e700 1 -- start start 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f3964108820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396359e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3964108d10 con 0x7f39640f52d0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f3964108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f3964108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44618/0 (socket says 172.21.15.114:44618) 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.163+0000 7f396259c700 1 -- 172.21.15.114:0/4171524106 learned_addr learned my addr 172.21.15.114:0/4171524106 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f396259c700 1 -- 172.21.15.114:0/4171524106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c005040 con 0x7f39640f52d0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f396259c700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f3964108820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f394c0089f0 tx=0x7f394c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c009320 con 0x7f39640f52d0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3964108f10 con 0x7f39640f52d0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f394c013ec0 con 0x7f39640f52d0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c014d30 con 0x7f39640f52d0 2024-02-20T13:24:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.164+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39641093a0 con 0x7f39640f52d0 2024-02-20T13:24:52.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.165+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f394c0263a0 con 0x7f39640f52d0 2024-02-20T13:24:52.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.165+0000 7f395affd700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3950031090 0x7f3950033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:52.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.165+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f394c042450 con 0x7f39640f52d0 2024-02-20T13:24:52.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.165+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39640591e0 con 0x7f39640f52d0 2024-02-20T13:24:52.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.165+0000 7f3961d9b700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3950031090 0x7f3950033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:52.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.166+0000 7f3961d9b700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3950031090 0x7f3950033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:52.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.170+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f394c0266d0 con 0x7f39640f52d0 2024-02-20T13:24:52.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.378+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3964045bb0 con 0x7f39640f52d0 2024-02-20T13:24:52.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.379+0000 7f395affd700 1 -- 172.21.15.114:0/4171524106 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f394c02b020 con 0x7f39640f52d0 2024-02-20T13:24:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:52.380 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3950031090 msgr2=0x7f3950033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3950031090 0x7f3950033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).stop 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 msgr2=0x7f3964108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f3964108820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f394c0089f0 tx=0x7f394c00b040).stop 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 shutdown_connections 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3950031090 0x7f3950033550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 --2- 172.21.15.114:0/4171524106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39640f52d0 0x7f3964108820 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 >> 172.21.15.114:0/4171524106 conn(0x7f39640f0a30 msgr2=0x7f396405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 shutdown_connections 2024-02-20T13:24:52.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:52.380+0000 7f396359e700 1 -- 172.21.15.114:0/4171524106 wait complete. 2024-02-20T13:24:52.383 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:53.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:52.207703+0000 mgr.smithi038.powepi (mgr.14156) 230 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:53.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:52.380334+0000 mon.smithi038 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.114:0/4171524106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:54.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:54.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:55.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:54.208113+0000 mgr.smithi038.powepi (mgr.14156) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:56.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1473438562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 msgr2=0x7f0cc00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:56.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 --2- 172.21.15.114:0/1473438562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc00f58a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f0cb0009fb0 tx=0x7f0cb0009b30).stop 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1473438562 shutdown_connections 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 --2- 172.21.15.114:0/1473438562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc00f58a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1473438562 >> 172.21.15.114:0/1473438562 conn(0x7f0cc00f0be0 msgr2=0x7f0cc00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1473438562 shutdown_connections 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.149+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1473438562 wait complete. 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cc4bb9700 1 Processor -- start 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cc4bb9700 1 -- start start 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cc4bb9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc01067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cc4bb9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0106ce0 con 0x7f0cc00f5480 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cbe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc01067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cbe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc01067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54406/0 (socket says 172.21.15.114:54406) 2024-02-20T13:24:56.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.150+0000 7f0cbe59c700 1 -- 172.21.15.114:0/1805957541 learned_addr learned my addr 172.21.15.114:0/1805957541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0cbe59c700 1 -- 172.21.15.114:0/1805957541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0005040 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0cbe59c700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc01067f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f0cb00089f0 tx=0x7f0cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb0009320 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0cb0013ec0 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb0014d30 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc0106ee0 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.151+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc0107370 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.152+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0cb00095f0 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.152+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc0045bb0 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.152+0000 7f0caeffd700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0ca8031090 0x7f0ca8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.152+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0cb00424a0 con 0x7f0cc00f5480 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.152+0000 7f0cbdd9b700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0ca8031090 0x7f0ca8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.153+0000 7f0cbdd9b700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0ca8031090 0x7f0ca8033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0cb40096a0 tx=0x7f0cb4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:24:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.157+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0cb001d7f0 con 0x7f0cc00f5480 2024-02-20T13:24:56.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.366+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cc00faf70 con 0x7f0cc00f5480 2024-02-20T13:24:56.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.368+0000 7f0caeffd700 1 -- 172.21.15.114:0/1805957541 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cb001d7f0 con 0x7f0cc00f5480 2024-02-20T13:24:56.369 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:24:56.369 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:24:56.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0ca8031090 msgr2=0x7f0ca8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0ca8031090 0x7f0ca8033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0cb40096a0 tx=0x7f0cb4006b40).stop 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 msgr2=0x7f0cc01067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc01067f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f0cb00089f0 tx=0x7f0cb000b040).stop 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 shutdown_connections 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0ca8031090 0x7f0ca8033550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 --2- 172.21.15.114:0/1805957541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cc00f5480 0x7f0cc01067f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 >> 172.21.15.114:0/1805957541 conn(0x7f0cc00f0be0 msgr2=0x7f0cc00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 shutdown_connections 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:24:56.369+0000 7f0cc4bb9700 1 -- 172.21.15.114:0/1805957541 wait complete. 2024-02-20T13:24:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:24:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:56.208531+0000 mgr.smithi038.powepi (mgr.14156) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:24:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:24:56.368768+0000 mon.smithi038 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.114:0/1805957541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:24:58.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:24:58.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:24:59.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:24:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:24:58.208972+0000 mgr.smithi038.powepi (mgr.14156) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:00.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 -- 172.21.15.114:0/2916361990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 msgr2=0x7feddc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 --2- 172.21.15.114:0/2916361990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc0f58a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fedd0009fb0 tx=0x7fedd0009b30).stop 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 -- 172.21.15.114:0/2916361990 shutdown_connections 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 --2- 172.21.15.114:0/2916361990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc0f58a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 -- 172.21.15.114:0/2916361990 >> 172.21.15.114:0/2916361990 conn(0x7feddc0f0be0 msgr2=0x7feddc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 -- 172.21.15.114:0/2916361990 shutdown_connections 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 -- 172.21.15.114:0/2916361990 wait complete. 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 Processor -- start 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.122+0000 7fede1013700 1 -- start start 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7fede1013700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7fede1013700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc106cd0 con 0x7feddc0f5480 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7feddad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7feddad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54428/0 (socket says 172.21.15.114:54428) 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7feddad9d700 1 -- 172.21.15.114:0/1101280644 learned_addr learned my addr 172.21.15.114:0/1101280644 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7feddad9d700 1 -- 172.21.15.114:0/1101280644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0005040 con 0x7feddc0f5480 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.123+0000 7feddad9d700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc1067e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fedd00089f0 tx=0x7fedd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.124+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedd0009320 con 0x7feddc0f5480 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.124+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fedd0013ec0 con 0x7feddc0f5480 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.124+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc106ed0 con 0x7feddc0f5480 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.124+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedd0014d30 con 0x7feddc0f5480 2024-02-20T13:25:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.124+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc107360 con 0x7feddc0f5480 2024-02-20T13:25:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.124+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fedd00263a0 con 0x7feddc0f5480 2024-02-20T13:25:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.125+0000 7fedc37fe700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fedc4031090 0x7fedc4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.125+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fedd00423d0 con 0x7feddc0f5480 2024-02-20T13:25:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.125+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc045bb0 con 0x7feddc0f5480 2024-02-20T13:25:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.125+0000 7fedda59c700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fedc4031090 0x7fedc4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.125+0000 7fedda59c700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fedc4031090 0x7fedc4033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fedcc0096a0 tx=0x7fedcc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:00.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.131+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fedd00266d0 con 0x7feddc0f5480 2024-02-20T13:25:00.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.337+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feddc107fe0 con 0x7feddc0f5480 2024-02-20T13:25:00.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.338+0000 7fedc37fe700 1 -- 172.21.15.114:0/1101280644 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fedd0012330 con 0x7feddc0f5480 2024-02-20T13:25:00.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:00.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fedc4031090 msgr2=0x7fedc4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:00.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fedc4031090 0x7fedc4033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fedcc0096a0 tx=0x7fedcc006b40).stop 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 msgr2=0x7feddc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc1067e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fedd00089f0 tx=0x7fedd000b040).stop 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 shutdown_connections 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fedc4031090 0x7fedc4033550 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 --2- 172.21.15.114:0/1101280644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feddc0f5480 0x7feddc1067e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 >> 172.21.15.114:0/1101280644 conn(0x7feddc0f0be0 msgr2=0x7feddc05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 shutdown_connections 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:00.340+0000 7fede1013700 1 -- 172.21.15.114:0/1101280644 wait complete. 2024-02-20T13:25:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:01.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:00.209492+0000 mgr.smithi038.powepi (mgr.14156) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:00.339789+0000 mon.smithi038 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.114:0/1101280644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:02.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:02.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:03.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:02.209948+0000 mgr.smithi038.powepi (mgr.14156) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 -- 172.21.15.114:0/837669563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 msgr2=0x7f5f9c0f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 --2- 172.21.15.114:0/837669563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c0f91f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5f90009fb0 tx=0x7f5f90009b30).stop 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 -- 172.21.15.114:0/837669563 shutdown_connections 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 --2- 172.21.15.114:0/837669563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c0f91f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 -- 172.21.15.114:0/837669563 >> 172.21.15.114:0/837669563 conn(0x7f5f9c0f0c10 msgr2=0x7f5f9c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 -- 172.21.15.114:0/837669563 shutdown_connections 2024-02-20T13:25:04.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.113+0000 7f5fa1f20700 1 -- 172.21.15.114:0/837669563 wait complete. 2024-02-20T13:25:04.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5fa1f20700 1 Processor -- start 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5fa1f20700 1 -- start start 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5fa1f20700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5fa1f20700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c106c80 con 0x7f5f9c060300 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5f9b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5f9b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54448/0 (socket says 172.21.15.114:54448) 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5f9b7fe700 1 -- 172.21.15.114:0/2607900677 learned_addr learned my addr 172.21.15.114:0/2607900677 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.114+0000 7f5f9b7fe700 1 -- 172.21.15.114:0/2607900677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90005040 con 0x7f5f9c060300 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5f9b7fe700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c106790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f5f90009f80 tx=0x7f5f9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f90009320 con 0x7f5f9c060300 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5f90013680 con 0x7f5f9c060300 2024-02-20T13:25:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c106e80 con 0x7f5f9c060300 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9001eda0 con 0x7f5f9c060300 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c107310 con 0x7f5f9c060300 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.115+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5f90014420 con 0x7f5f9c060300 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.116+0000 7f5f83fff700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5f840310e0 0x7f5f840335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.116+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5f900423c0 con 0x7f5f9c060300 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.116+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c045bb0 con 0x7f5f9c060300 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.116+0000 7f5f9affd700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5f840310e0 0x7f5f840335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.116+0000 7f5f9affd700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5f840310e0 0x7f5f840335a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5f8c0096a0 tx=0x7f5f8c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:04.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.120+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5f900146b0 con 0x7f5f9c060300 2024-02-20T13:25:04.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.327+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f9c0fc180 con 0x7f5f9c060300 2024-02-20T13:25:04.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.328+0000 7f5f83fff700 1 -- 172.21.15.114:0/2607900677 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f90026350 con 0x7f5f9c060300 2024-02-20T13:25:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:04.329 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:04.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5f840310e0 msgr2=0x7f5f840335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5f840310e0 0x7f5f840335a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5f8c0096a0 tx=0x7f5f8c006b40).stop 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 msgr2=0x7f5f9c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c106790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f5f90009f80 tx=0x7f5f9000b040).stop 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 shutdown_connections 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5f840310e0 0x7f5f840335a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.329+0000 7f5fa1f20700 1 --2- 172.21.15.114:0/2607900677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9c060300 0x7f5f9c106790 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.330+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 >> 172.21.15.114:0/2607900677 conn(0x7f5f9c0f0c10 msgr2=0x7f5f9c0f16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.330+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 shutdown_connections 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:04.330+0000 7f5fa1f20700 1 -- 172.21.15.114:0/2607900677 wait complete. 2024-02-20T13:25:04.331 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:25:03.736739+0000 mgr.smithi038.powepi (mgr.14156) 236 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-node-exporter.smithi038 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-node-exporter.smithi038 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon node-exporter.smithi038 ... 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Verifying port 9100 ... 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 1 from systemctl start ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: systemctl: stderr Job for ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038.service failed because a timeout was exceeded. 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: systemctl: stderr See "systemctl status ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038.service" and "journalctl -xe" for details. 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:25:05.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ports=daemon_ports) 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: Failed command: systemctl start ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-02-20T13:25:05.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: yield (conn, connr) 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: code, '\n'.join(err))) 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-node-exporter.smithi038 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-node-exporter.smithi038 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon node-exporter.smithi038 ... 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Verifying port 9100 ... 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 1 from systemctl start ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: systemctl: stderr Job for ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038.service failed because a timeout was exceeded. 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: systemctl: stderr See "systemctl status ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038.service" and "journalctl -xe" for details. 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:25:05.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ports=daemon_ports) 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: Failed command: systemctl start ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:25:03.736954+0000 mgr.smithi038.powepi (mgr.14156) 237 : cephadm [ERR] Failed while placing node-exporter.smithi038 on smithi038: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-node-exporter.smithi038 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-node-exporter.smithi038 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon node-exporter.smithi038 ... 2024-02-20T13:25:05.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Verifying port 9100 ... 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 1 from systemctl start ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: systemctl: stderr Job for ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038.service failed because a timeout was exceeded. 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: systemctl: stderr See "systemctl status ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038.service" and "journalctl -xe" for details. 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: ports=daemon_ports) 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T13:25:05.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: Failed command: systemctl start ceph-24090638-cff2-11ee-95bb-87774f69a715@node-exporter.smithi038 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:25:03.743056+0000 mgr.smithi038.powepi (mgr.14156) 238 : cephadm [INF] Deploying daemon prometheus.smithi038 on smithi038 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:04.210252+0000 mgr.smithi038.powepi (mgr.14156) 239 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:05.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:04.328351+0000 mon.smithi038 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.114:0/2607900677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:05.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:05.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:07.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:06.210660+0000 mgr.smithi038.powepi (mgr.14156) 240 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:07.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 -- 172.21.15.114:0/1332759017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 msgr2=0x7f45fc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 --2- 172.21.15.114:0/1332759017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc0f56f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f45ec009fb0 tx=0x7f45ec009b30).stop 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 -- 172.21.15.114:0/1332759017 shutdown_connections 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 --2- 172.21.15.114:0/1332759017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc0f56f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 -- 172.21.15.114:0/1332759017 >> 172.21.15.114:0/1332759017 conn(0x7f45fc0f0a30 msgr2=0x7f45fc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 -- 172.21.15.114:0/1332759017 shutdown_connections 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.965+0000 7f45fbfff700 1 -- 172.21.15.114:0/1332759017 wait complete. 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45fbfff700 1 Processor -- start 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45fbfff700 1 -- start start 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45fbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45fbfff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc106af0 con 0x7f45fc0f52d0 2024-02-20T13:25:07.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45faffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45faffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50042/0 (socket says 172.21.15.114:50042) 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45faffd700 1 -- 172.21.15.114:0/2489757145 learned_addr learned my addr 172.21.15.114:0/2489757145 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.966+0000 7f45faffd700 1 -- 172.21.15.114:0/2489757145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec005040 con 0x7f45fc0f52d0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.967+0000 7f45faffd700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc106600 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f45ec0089f0 tx=0x7f45ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.967+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec009320 con 0x7f45fc0f52d0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.967+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc106cf0 con 0x7f45fc0f52d0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.967+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f45ec013ec0 con 0x7f45fc0f52d0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.967+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec014d30 con 0x7f45fc0f52d0 2024-02-20T13:25:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.967+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc107180 con 0x7f45fc0f52d0 2024-02-20T13:25:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.968+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f45ec014420 con 0x7f45fc0f52d0 2024-02-20T13:25:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.968+0000 7f45e37fe700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f45e4031090 0x7f45e4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.968+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f45ec0423e0 con 0x7f45fc0f52d0 2024-02-20T13:25:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.968+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc045bb0 con 0x7f45fc0f52d0 2024-02-20T13:25:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.968+0000 7f45fa7fc700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f45e4031090 0x7f45e4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.968+0000 7f45fa7fc700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f45e4031090 0x7f45e4033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f45f00096a0 tx=0x7f45f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:07.973+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f45ec014750 con 0x7f45fc0f52d0 2024-02-20T13:25:08.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.180+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45fc0faeb0 con 0x7f45fc0f52d0 2024-02-20T13:25:08.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.181+0000 7f45e37fe700 1 -- 172.21.15.114:0/2489757145 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45ec01d7f0 con 0x7f45fc0f52d0 2024-02-20T13:25:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:08.182 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f45e4031090 msgr2=0x7f45e4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f45e4031090 0x7f45e4033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f45f00096a0 tx=0x7f45f0006b40).stop 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 msgr2=0x7f45fc106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc106600 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f45ec0089f0 tx=0x7f45ec00b040).stop 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 shutdown_connections 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f45e4031090 0x7f45e4033550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 --2- 172.21.15.114:0/2489757145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0f52d0 0x7f45fc106600 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 >> 172.21.15.114:0/2489757145 conn(0x7f45fc0f0a30 msgr2=0x7f45fc05fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 shutdown_connections 2024-02-20T13:25:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:08.183+0000 7f45fbfff700 1 -- 172.21.15.114:0/2489757145 wait complete. 2024-02-20T13:25:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:08.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:08.181779+0000 mon.smithi038 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.114:0/2489757145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:09.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:08.211036+0000 mgr.smithi038.powepi (mgr.14156) 241 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:08.294386+0000 mon.smithi038 (mon.0) 245 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:25:09.993 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:09.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:11.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:10.211498+0000 mgr.smithi038.powepi (mgr.14156) 242 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:12.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 -- 172.21.15.114:0/1956979125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 msgr2=0x7fd7880f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:12.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 --2- 172.21.15.114:0/1956979125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd7880f6800 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd784009fb0 tx=0x7fd784009b30).stop 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 -- 172.21.15.114:0/1956979125 shutdown_connections 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 --2- 172.21.15.114:0/1956979125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd7880f6800 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 -- 172.21.15.114:0/1956979125 >> 172.21.15.114:0/1956979125 conn(0x7fd7880f1b40 msgr2=0x7fd7880f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 -- 172.21.15.114:0/1956979125 shutdown_connections 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 -- 172.21.15.114:0/1956979125 wait complete. 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 Processor -- start 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.070+0000 7fd78f0c2700 1 -- start start 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.071+0000 7fd78f0c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd788106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.071+0000 7fd78f0c2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd788106b10 con 0x7fd7880f63e0 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.071+0000 7fd78e0c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd788106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.071+0000 7fd78e0c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd788106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50058/0 (socket says 172.21.15.114:50058) 2024-02-20T13:25:12.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.071+0000 7fd78e0c0700 1 -- 172.21.15.114:0/2164155080 learned_addr learned my addr 172.21.15.114:0/2164155080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.071+0000 7fd78e0c0700 1 -- 172.21.15.114:0/2164155080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784005040 con 0x7fd7880f63e0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.072+0000 7fd78e0c0700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd788106620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fd7840089f0 tx=0x7fd78400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.072+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd784009320 con 0x7fd7880f63e0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.072+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd788106d10 con 0x7fd7880f63e0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.072+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd784013ec0 con 0x7fd7880f63e0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.072+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd784014d30 con 0x7fd7880f63e0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.072+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7881071a0 con 0x7fd7880f63e0 2024-02-20T13:25:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.073+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd7840095f0 con 0x7fd7880f63e0 2024-02-20T13:25:12.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.073+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd788045bb0 con 0x7fd7880f63e0 2024-02-20T13:25:12.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.073+0000 7fd77effd700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd774031090 0x7fd774033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:12.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.073+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd784042460 con 0x7fd7880f63e0 2024-02-20T13:25:12.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.073+0000 7fd78d8bf700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd774031090 0x7fd774033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:12.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.074+0000 7fd78d8bf700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd774031090 0x7fd774033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd7780096a0 tx=0x7fd778006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.078+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd78401d7f0 con 0x7fd7880f63e0 2024-02-20T13:25:12.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.286+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd788108050 con 0x7fd7880f63e0 2024-02-20T13:25:12.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.287+0000 7fd77effd700 1 -- 172.21.15.114:0/2164155080 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd78401d7f0 con 0x7fd7880f63e0 2024-02-20T13:25:12.288 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:12.288 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:12.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd774031090 msgr2=0x7fd774033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:12.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd774031090 0x7fd774033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd7780096a0 tx=0x7fd778006b40).stop 2024-02-20T13:25:12.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 msgr2=0x7fd788106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:12.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd788106620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fd7840089f0 tx=0x7fd78400b040).stop 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 shutdown_connections 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd774031090 0x7fd774033550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 --2- 172.21.15.114:0/2164155080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7880f63e0 0x7fd788106620 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 >> 172.21.15.114:0/2164155080 conn(0x7fd7880f1b40 msgr2=0x7fd7880f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 shutdown_connections 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:12.289+0000 7fd78f0c2700 1 -- 172.21.15.114:0/2164155080 wait complete. 2024-02-20T13:25:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:13.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:12.211990+0000 mgr.smithi038.powepi (mgr.14156) 243 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:13.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:12.287733+0000 mon.smithi038 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.114:0/2164155080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:14.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:14.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:15.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:14.212478+0000 mgr.smithi038.powepi (mgr.14156) 244 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 -- 172.21.15.114:0/693831056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 msgr2=0x7f0ab40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 --2- 172.21.15.114:0/693831056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40f56f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f0ab0009fb0 tx=0x7f0ab0009b30).stop 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 -- 172.21.15.114:0/693831056 shutdown_connections 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 --2- 172.21.15.114:0/693831056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40f56f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 -- 172.21.15.114:0/693831056 >> 172.21.15.114:0/693831056 conn(0x7f0ab40f0a30 msgr2=0x7f0ab40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 -- 172.21.15.114:0/693831056 shutdown_connections 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.876+0000 7f0abac0a700 1 -- 172.21.15.114:0/693831056 wait complete. 2024-02-20T13:25:15.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0abac0a700 1 Processor -- start 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0abac0a700 1 -- start start 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0abac0a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0abac0a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40fb730 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0ab9c08700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0ab9c08700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58060/0 (socket says 172.21.15.114:58060) 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0ab9c08700 1 -- 172.21.15.114:0/753732791 learned_addr learned my addr 172.21.15.114:0/753732791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.877+0000 7f0ab9c08700 1 -- 172.21.15.114:0/753732791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0005040 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.878+0000 7f0ab9c08700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40fd2b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f0ab0000c00 tx=0x7f0ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.878+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab00095f0 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.878+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab40fb8d0 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.878+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0ab0007e50 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.878+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab0014e70 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.878+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab40fbd60 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.879+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0ab00146b0 con 0x7f0ab40f52d0 2024-02-20T13:25:15.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.879+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab40591e0 con 0x7f0ab40f52d0 2024-02-20T13:25:15.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.879+0000 7f0aaa7fc700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0aa00310e0 0x7f0aa00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:15.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.879+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0ab0041ca0 con 0x7f0ab40f52d0 2024-02-20T13:25:15.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.879+0000 7f0ab9407700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0aa00310e0 0x7f0aa00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:15.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.880+0000 7f0ab9407700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0aa00310e0 0x7f0aa00335a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f0aa40096a0 tx=0x7f0aa4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:15.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:15.884+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0ab001b2c0 con 0x7f0ab40f52d0 2024-02-20T13:25:16.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.089+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ab4045bb0 con 0x7f0ab40f52d0 2024-02-20T13:25:16.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.090+0000 7f0aaa7fc700 1 -- 172.21.15.114:0/753732791 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ab001b2c0 con 0x7f0ab40f52d0 2024-02-20T13:25:16.091 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:16.091 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0aa00310e0 msgr2=0x7f0aa00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0aa00310e0 0x7f0aa00335a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f0aa40096a0 tx=0x7f0aa4006b40).stop 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 msgr2=0x7f0ab40fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40fd2b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f0ab0000c00 tx=0x7f0ab000b040).stop 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 shutdown_connections 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0aa00310e0 0x7f0aa00335a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 --2- 172.21.15.114:0/753732791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab40f52d0 0x7f0ab40fd2b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 >> 172.21.15.114:0/753732791 conn(0x7f0ab40f0a30 msgr2=0x7f0ab405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:16.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 shutdown_connections 2024-02-20T13:25:16.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:16.091+0000 7f0abac0a700 1 -- 172.21.15.114:0/753732791 wait complete. 2024-02-20T13:25:16.093 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:16.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:16.090259+0000 mon.smithi038 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.114:0/753732791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:17.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:16.212890+0000 mgr.smithi038.powepi (mgr.14156) 245 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:17.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:17.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:19.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:18.213232+0000 mgr.smithi038.powepi (mgr.14156) 246 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:19.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 -- 172.21.15.114:0/2144218567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 msgr2=0x7f0f5c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 --2- 172.21.15.114:0/2144218567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c0f69b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f0f4c004760 tx=0x7f0f4c009ad0).stop 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 -- 172.21.15.114:0/2144218567 shutdown_connections 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 --2- 172.21.15.114:0/2144218567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c0f69b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 -- 172.21.15.114:0/2144218567 >> 172.21.15.114:0/2144218567 conn(0x7f0f5c0f1cd0 msgr2=0x7f0f5c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 -- 172.21.15.114:0/2144218567 shutdown_connections 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.730+0000 7f0f63f16700 1 -- 172.21.15.114:0/2144218567 wait complete. 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f63f16700 1 Processor -- start 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f63f16700 1 -- start start 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f63f16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f63f16700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c069f00 con 0x7f0f5c0f6590 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f61cb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:19.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f61cb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58074/0 (socket says 172.21.15.114:58074) 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f61cb2700 1 -- 172.21.15.114:0/1376484434 learned_addr learned my addr 172.21.15.114:0/1376484434 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.731+0000 7f0f61cb2700 1 -- 172.21.15.114:0/1376484434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c009d70 con 0x7f0f5c0f6590 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.732+0000 7f0f61cb2700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c06ba80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0f4c013f10 tx=0x7f0f4c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.732+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4c00e040 con 0x7f0f5c0f6590 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.732+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c06a0a0 con 0x7f0f5c0f6590 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.732+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0f4c007df0 con 0x7f0f5c0f6590 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.732+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4c014d60 con 0x7f0f5c0f6590 2024-02-20T13:25:19.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.732+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c06a530 con 0x7f0f5c0f6590 2024-02-20T13:25:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.733+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0f4c014520 con 0x7f0f5c0f6590 2024-02-20T13:25:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.733+0000 7f0f527fc700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0f48031090 0x7f0f48033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.733+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c0591e0 con 0x7f0f5c0f6590 2024-02-20T13:25:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.733+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0f4c042490 con 0x7f0f5c0f6590 2024-02-20T13:25:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.733+0000 7f0f614b1700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0f48031090 0x7f0f48033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.733+0000 7f0f614b1700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0f48031090 0x7f0f48033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f0f580096a0 tx=0x7f0f58006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:19.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.738+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0f4c007960 con 0x7f0f5c0f6590 2024-02-20T13:25:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.944+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f5c06acd0 con 0x7f0f5c0f6590 2024-02-20T13:25:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.945+0000 7f0f527fc700 1 -- 172.21.15.114:0/1376484434 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f4c026760 con 0x7f0f5c0f6590 2024-02-20T13:25:19.946 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:19.946 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0f48031090 msgr2=0x7f0f48033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0f48031090 0x7f0f48033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f0f580096a0 tx=0x7f0f58006b40).stop 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 msgr2=0x7f0f5c06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c06ba80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0f4c013f10 tx=0x7f0f4c009ad0).stop 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 shutdown_connections 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0f48031090 0x7f0f48033550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 --2- 172.21.15.114:0/1376484434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f5c0f6590 0x7f0f5c06ba80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:19.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 >> 172.21.15.114:0/1376484434 conn(0x7f0f5c0f1cd0 msgr2=0x7f0f5c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:19.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 shutdown_connections 2024-02-20T13:25:19.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:19.946+0000 7f0f63f16700 1 -- 172.21.15.114:0/1376484434 wait complete. 2024-02-20T13:25:19.948 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:20.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:19.945545+0000 mon.smithi038 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.114:0/1376484434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:21.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:21.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:21.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:20.213580+0000 mgr.smithi038.powepi (mgr.14156) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:23.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 -- 172.21.15.114:0/1643002938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 msgr2=0x7f82880f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 --2- 172.21.15.114:0/1643002938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82880f69b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f827c009fb0 tx=0x7f827c009b30).stop 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 -- 172.21.15.114:0/1643002938 shutdown_connections 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 --2- 172.21.15.114:0/1643002938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82880f69b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 -- 172.21.15.114:0/1643002938 >> 172.21.15.114:0/1643002938 conn(0x7f82880f1cf0 msgr2=0x7f82880f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 -- 172.21.15.114:0/1643002938 shutdown_connections 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.608+0000 7f828d23e700 1 -- 172.21.15.114:0/1643002938 wait complete. 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f828d23e700 1 Processor -- start 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f828d23e700 1 -- start start 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f828d23e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82881067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f828d23e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8288106cc0 con 0x7f82880f6590 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f8286d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82881067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f8286d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82881067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58100/0 (socket says 172.21.15.114:58100) 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f8286d9d700 1 -- 172.21.15.114:0/3295159700 learned_addr learned my addr 172.21.15.114:0/3295159700 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:23.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f8286d9d700 1 -- 172.21.15.114:0/3295159700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c005040 con 0x7f82880f6590 2024-02-20T13:25:23.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.609+0000 7f8286d9d700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82881067d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f827c0089f0 tx=0x7f827c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:23.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.610+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827c009320 con 0x7f82880f6590 2024-02-20T13:25:23.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.610+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f827c013ec0 con 0x7f82880f6590 2024-02-20T13:25:23.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.610+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8288106ec0 con 0x7f82880f6590 2024-02-20T13:25:23.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.610+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827c014d30 con 0x7f82880f6590 2024-02-20T13:25:23.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.610+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8288107350 con 0x7f82880f6590 2024-02-20T13:25:23.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.610+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f827c0095f0 con 0x7f82880f6590 2024-02-20T13:25:23.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.611+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8288045bb0 con 0x7f82880f6590 2024-02-20T13:25:23.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.611+0000 7f826f7fe700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8270031090 0x7f8270033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:23.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.611+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f827c0423e0 con 0x7f82880f6590 2024-02-20T13:25:23.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.611+0000 7f828659c700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8270031090 0x7f8270033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:23.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.611+0000 7f828659c700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8270031090 0x7f8270033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f82780096a0 tx=0x7f8278006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:23.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.616+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f827c01d7f0 con 0x7f82880f6590 2024-02-20T13:25:23.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:22.213963+0000 mgr.smithi038.powepi (mgr.14156) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:23.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.824+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8288108200 con 0x7f82880f6590 2024-02-20T13:25:23.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.824+0000 7f826f7fe700 1 -- 172.21.15.114:0/3295159700 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f827c01d7f0 con 0x7f82880f6590 2024-02-20T13:25:23.825 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:23.826 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8270031090 msgr2=0x7f8270033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8270031090 0x7f8270033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f82780096a0 tx=0x7f8278006b40).stop 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 msgr2=0x7f82881067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82881067d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f827c0089f0 tx=0x7f827c00b040).stop 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 shutdown_connections 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f8270031090 0x7f8270033550 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 --2- 172.21.15.114:0/3295159700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82880f6590 0x7f82881067d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 >> 172.21.15.114:0/3295159700 conn(0x7f82880f1cf0 msgr2=0x7f82880f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 shutdown_connections 2024-02-20T13:25:23.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:23.826+0000 7f828d23e700 1 -- 172.21.15.114:0/3295159700 wait complete. 2024-02-20T13:25:23.828 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:24.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:23.825133+0000 mon.smithi038 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.114:0/3295159700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:25.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:25.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:24.214256+0000 mgr.smithi038.powepi (mgr.14156) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:27.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 -- 172.21.15.114:0/2733726224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 msgr2=0x7f302c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 --2- 172.21.15.114:0/2733726224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c0f69b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f301c004760 tx=0x7f301c009ad0).stop 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 -- 172.21.15.114:0/2733726224 shutdown_connections 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 --2- 172.21.15.114:0/2733726224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c0f69b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 -- 172.21.15.114:0/2733726224 >> 172.21.15.114:0/2733726224 conn(0x7f302c0f1cd0 msgr2=0x7f302c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 -- 172.21.15.114:0/2733726224 shutdown_connections 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.558+0000 7f3033e86700 1 -- 172.21.15.114:0/2733726224 wait complete. 2024-02-20T13:25:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3033e86700 1 Processor -- start 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3033e86700 1 -- start start 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3033e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3033e86700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c106cb0 con 0x7f302c0f6590 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3031c22700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3031c22700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:49768/0 (socket says 172.21.15.114:49768) 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3031c22700 1 -- 172.21.15.114:0/2028194675 learned_addr learned my addr 172.21.15.114:0/2028194675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3031c22700 1 -- 172.21.15.114:0/2028194675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009d70 con 0x7f302c0f6590 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.559+0000 7f3031c22700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c1067c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f301c013f10 tx=0x7f301c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.560+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f301c00e040 con 0x7f302c0f6590 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.560+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c106eb0 con 0x7f302c0f6590 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.560+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f301c007df0 con 0x7f302c0f6590 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.560+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f301c014d60 con 0x7f302c0f6590 2024-02-20T13:25:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.560+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c107340 con 0x7f302c0f6590 2024-02-20T13:25:27.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.560+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f301c014520 con 0x7f302c0f6590 2024-02-20T13:25:27.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.561+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c045bb0 con 0x7f302c0f6590 2024-02-20T13:25:27.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.561+0000 7f30227fc700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f30180310e0 0x7f30180335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:27.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.561+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f301c042580 con 0x7f302c0f6590 2024-02-20T13:25:27.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.561+0000 7f3031421700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f30180310e0 0x7f30180335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:27.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.562+0000 7f3031421700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f30180310e0 0x7f30180335a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f30280096a0 tx=0x7f3028006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.566+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f301c007960 con 0x7f302c0f6590 2024-02-20T13:25:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:26.214684+0000 mgr.smithi038.powepi (mgr.14156) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:27.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.771+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f302c1081f0 con 0x7f302c0f6590 2024-02-20T13:25:27.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.772+0000 7f30227fc700 1 -- 172.21.15.114:0/2028194675 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f301c0262f0 con 0x7f302c0f6590 2024-02-20T13:25:27.773 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:27.773 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.773+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f30180310e0 msgr2=0x7f30180335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.773+0000 7f3033e86700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f30180310e0 0x7f30180335a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f30280096a0 tx=0x7f3028006b40).stop 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.773+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 msgr2=0x7f302c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.773+0000 7f3033e86700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c1067c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f301c013f10 tx=0x7f301c009ad0).stop 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.774+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 shutdown_connections 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.774+0000 7f3033e86700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f30180310e0 0x7f30180335a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.774+0000 7f3033e86700 1 --2- 172.21.15.114:0/2028194675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f302c0f6590 0x7f302c1067c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:27.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.774+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 >> 172.21.15.114:0/2028194675 conn(0x7f302c0f1cd0 msgr2=0x7f302c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:27.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.774+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 shutdown_connections 2024-02-20T13:25:27.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:27.774+0000 7f3033e86700 1 -- 172.21.15.114:0/2028194675 wait complete. 2024-02-20T13:25:27.777 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:27.773451+0000 mon.smithi038 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.114:0/2028194675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:29.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:29.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:28.215155+0000 mgr.smithi038.powepi (mgr.14156) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:28.282598+0000 mon.smithi038 (mon.0) 251 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:25:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:28.282832+0000 mon.smithi038 (mon.0) 252 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:25:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 -- 172.21.15.114:0/2003339309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 msgr2=0x7fd5c00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 --2- 172.21.15.114:0/2003339309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c00f58d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fd5b0009fb0 tx=0x7fd5b0009b30).stop 2024-02-20T13:25:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 -- 172.21.15.114:0/2003339309 shutdown_connections 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 --2- 172.21.15.114:0/2003339309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c00f58d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 -- 172.21.15.114:0/2003339309 >> 172.21.15.114:0/2003339309 conn(0x7fd5c00f0c10 msgr2=0x7fd5c00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 -- 172.21.15.114:0/2003339309 shutdown_connections 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 -- 172.21.15.114:0/2003339309 wait complete. 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 Processor -- start 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.397+0000 7fd5c5386700 1 -- start start 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5c5386700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c0106840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5c5386700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c0106d30 con 0x7fd5c00f54b0 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5beffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5beffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:49788/0 (socket says 172.21.15.114:49788) 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5beffd700 1 -- 172.21.15.114:0/1570372511 learned_addr learned my addr 172.21.15.114:0/1570372511 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5beffd700 1 -- 172.21.15.114:0/1570372511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b0005040 con 0x7fd5c00f54b0 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.398+0000 7fd5beffd700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c0106840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fd5b0009f80 tx=0x7fd5b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.399+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b0009320 con 0x7fd5c00f54b0 2024-02-20T13:25:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.399+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c0106f30 con 0x7fd5c00f54b0 2024-02-20T13:25:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.399+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd5b0013380 con 0x7fd5c00f54b0 2024-02-20T13:25:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.399+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b001ebd0 con 0x7fd5c00f54b0 2024-02-20T13:25:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.399+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c01073c0 con 0x7fd5c00f54b0 2024-02-20T13:25:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.399+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd5b001e420 con 0x7fd5c00f54b0 2024-02-20T13:25:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.400+0000 7fd5a77fe700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd5a8031090 0x7fd5a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.400+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c0045bb0 con 0x7fd5c00f54b0 2024-02-20T13:25:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.400+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd5b0042220 con 0x7fd5c00f54b0 2024-02-20T13:25:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.400+0000 7fd5be7fc700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd5a8031090 0x7fd5a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.400+0000 7fd5be7fc700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd5a8031090 0x7fd5a8033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd5b40096a0 tx=0x7fd5b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.405+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd5b001c920 con 0x7fd5c00f54b0 2024-02-20T13:25:31.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.611+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5c00fafa0 con 0x7fd5c00f54b0 2024-02-20T13:25:31.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.612+0000 7fd5a77fe700 1 -- 172.21.15.114:0/1570372511 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5b001ee60 con 0x7fd5c00f54b0 2024-02-20T13:25:31.614 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:31.614 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:31.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd5a8031090 msgr2=0x7fd5a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:31.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd5a8031090 0x7fd5a8033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd5b40096a0 tx=0x7fd5b4006b40).stop 2024-02-20T13:25:31.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 msgr2=0x7fd5c0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:31.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c0106840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fd5b0009f80 tx=0x7fd5b000b040).stop 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 shutdown_connections 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fd5a8031090 0x7fd5a8033550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 --2- 172.21.15.114:0/1570372511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c00f54b0 0x7fd5c0106840 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 >> 172.21.15.114:0/1570372511 conn(0x7fd5c00f0c10 msgr2=0x7fd5c005ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 shutdown_connections 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:31.614+0000 7fd5c5386700 1 -- 172.21.15.114:0/1570372511 wait complete. 2024-02-20T13:25:31.616 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:30.215632+0000 mgr.smithi038.powepi (mgr.14156) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:32.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:31.613923+0000 mon.smithi038 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.114:0/1570372511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:33.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:33.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:33.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:32.216041+0000 mgr.smithi038.powepi (mgr.14156) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 -- 172.21.15.114:0/938532143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 msgr2=0x7fb8880f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 --2- 172.21.15.114:0/938532143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8880f58a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb884009fb0 tx=0x7fb884009b30).stop 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 -- 172.21.15.114:0/938532143 shutdown_connections 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 --2- 172.21.15.114:0/938532143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8880f58a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 -- 172.21.15.114:0/938532143 >> 172.21.15.114:0/938532143 conn(0x7fb8880f0be0 msgr2=0x7fb8880f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 -- 172.21.15.114:0/938532143 shutdown_connections 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.400+0000 7fb88f49e700 1 -- 172.21.15.114:0/938532143 wait complete. 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88f49e700 1 Processor -- start 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88f49e700 1 -- start start 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88f49e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8881056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88f49e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb888105be0 con 0x7fb8880f5480 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88d23a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8881056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88d23a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8881056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50988/0 (socket says 172.21.15.114:50988) 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88d23a700 1 -- 172.21.15.114:0/1125031021 learned_addr learned my addr 172.21.15.114:0/1125031021 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.401+0000 7fb88d23a700 1 -- 172.21.15.114:0/1125031021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb884005040 con 0x7fb8880f5480 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.402+0000 7fb88d23a700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8881056f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fb8840089f0 tx=0x7fb88400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.402+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb884009320 con 0x7fb8880f5480 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.402+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb888105de0 con 0x7fb8880f5480 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.402+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb884013ec0 con 0x7fb8880f5480 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.402+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb884014d30 con 0x7fb8880f5480 2024-02-20T13:25:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.402+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb888106270 con 0x7fb8880f5480 2024-02-20T13:25:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.403+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb884014420 con 0x7fb8880f5480 2024-02-20T13:25:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.403+0000 7fb87dffb700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb874031090 0x7fb874033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.403+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb884042400 con 0x7fb8880f5480 2024-02-20T13:25:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.403+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb888045bb0 con 0x7fb8880f5480 2024-02-20T13:25:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.403+0000 7fb88ca39700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb874031090 0x7fb874033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.404+0000 7fb88ca39700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb874031090 0x7fb874033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb8780096a0 tx=0x7fb878006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.408+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb884014750 con 0x7fb8880f5480 2024-02-20T13:25:35.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.617+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8880fb060 con 0x7fb8880f5480 2024-02-20T13:25:35.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.619+0000 7fb87dffb700 1 -- 172.21.15.114:0/1125031021 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb88401d380 con 0x7fb8880f5480 2024-02-20T13:25:35.620 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:35.620 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb874031090 msgr2=0x7fb874033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb874031090 0x7fb874033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb8780096a0 tx=0x7fb878006b40).stop 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 msgr2=0x7fb8881056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8881056f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fb8840089f0 tx=0x7fb88400b040).stop 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 shutdown_connections 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb874031090 0x7fb874033550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 --2- 172.21.15.114:0/1125031021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8880f5480 0x7fb8881056f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 >> 172.21.15.114:0/1125031021 conn(0x7fb8880f0be0 msgr2=0x7fb8880f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 shutdown_connections 2024-02-20T13:25:35.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:35.620+0000 7fb88f49e700 1 -- 172.21.15.114:0/1125031021 wait complete. 2024-02-20T13:25:35.623 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:35.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:34.216468+0000 mgr.smithi038.powepi (mgr.14156) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:36.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:35.619684+0000 mon.smithi038 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.114:0/1125031021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:37.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:37.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:37.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:36.216876+0000 mgr.smithi038.powepi (mgr.14156) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:39.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- 172.21.15.114:0/2670183800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 msgr2=0x7fdfc00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:39.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 --2- 172.21.15.114:0/2670183800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc00f69b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fdfbc004760 tx=0x7fdfbc009ad0).stop 2024-02-20T13:25:39.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- 172.21.15.114:0/2670183800 shutdown_connections 2024-02-20T13:25:39.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 --2- 172.21.15.114:0/2670183800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc00f69b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:39.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- 172.21.15.114:0/2670183800 >> 172.21.15.114:0/2670183800 conn(0x7fdfc00f1cd0 msgr2=0x7fdfc00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- 172.21.15.114:0/2670183800 shutdown_connections 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- 172.21.15.114:0/2670183800 wait complete. 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 Processor -- start 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- start start 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.411+0000 7fdfc8095700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc0106cb0 con 0x7fdfc00f6590 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc5e31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc5e31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:51006/0 (socket says 172.21.15.114:51006) 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc5e31700 1 -- 172.21.15.114:0/397023205 learned_addr learned my addr 172.21.15.114:0/397023205 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc5e31700 1 -- 172.21.15.114:0/397023205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc009d70 con 0x7fdfc00f6590 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc5e31700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc01067c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fdfbc004170 tx=0x7fdfbc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfbc00e040 con 0x7fdfc00f6590 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc0106eb0 con 0x7fdfc00f6590 2024-02-20T13:25:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdfbc008ef0 con 0x7fdfc00f6590 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfbc014c90 con 0x7fdfc00f6590 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.412+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc0107340 con 0x7fdfc00f6590 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.413+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdfbc014420 con 0x7fdfc00f6590 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.413+0000 7fdfb67fc700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdfac031090 0x7fdfac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.413+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc0045bb0 con 0x7fdfc00f6590 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.413+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdfbc0422d0 con 0x7fdfc00f6590 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.414+0000 7fdfc5630700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdfac031090 0x7fdfac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.414+0000 7fdfc5630700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdfac031090 0x7fdfac033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fdfb00096a0 tx=0x7fdfb0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.418+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdfbc026300 con 0x7fdfc00f6590 2024-02-20T13:25:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.627+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfc0108150 con 0x7fdfc00f6590 2024-02-20T13:25:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.628+0000 7fdfb67fc700 1 -- 172.21.15.114:0/397023205 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfbc012b20 con 0x7fdfc00f6590 2024-02-20T13:25:39.630 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:39.630 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:39.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdfac031090 msgr2=0x7fdfac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:39.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdfac031090 0x7fdfac033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fdfb00096a0 tx=0x7fdfb0006b40).stop 2024-02-20T13:25:39.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 msgr2=0x7fdfc01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:39.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc01067c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fdfbc004170 tx=0x7fdfbc009ad0).stop 2024-02-20T13:25:39.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 shutdown_connections 2024-02-20T13:25:39.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdfac031090 0x7fdfac033550 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:39.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 --2- 172.21.15.114:0/397023205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfc00f6590 0x7fdfc01067c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:39.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 >> 172.21.15.114:0/397023205 conn(0x7fdfc00f1cd0 msgr2=0x7fdfc00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:39.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 shutdown_connections 2024-02-20T13:25:39.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:39.629+0000 7fdfc8095700 1 -- 172.21.15.114:0/397023205 wait complete. 2024-02-20T13:25:39.632 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:39.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:38.217351+0000 mgr.smithi038.powepi (mgr.14156) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:40.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:39.629504+0000 mon.smithi038 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.114:0/397023205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:41.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:41.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:40.217816+0000 mgr.smithi038.powepi (mgr.14156) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 -- 172.21.15.114:0/3175454296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 msgr2=0x7ff5dc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 --2- 172.21.15.114:0/3175454296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc0f56f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff5cc009fb0 tx=0x7ff5cc009b30).stop 2024-02-20T13:25:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 -- 172.21.15.114:0/3175454296 shutdown_connections 2024-02-20T13:25:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 --2- 172.21.15.114:0/3175454296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc0f56f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 -- 172.21.15.114:0/3175454296 >> 172.21.15.114:0/3175454296 conn(0x7ff5dc0f0a30 msgr2=0x7ff5dc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 -- 172.21.15.114:0/3175454296 shutdown_connections 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.536+0000 7ff5e25a1700 1 -- 172.21.15.114:0/3175454296 wait complete. 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e25a1700 1 Processor -- start 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e25a1700 1 -- start start 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e25a1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e25a1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc060300 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e159f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e159f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:51030/0 (socket says 172.21.15.114:51030) 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.537+0000 7ff5e159f700 1 -- 172.21.15.114:0/285666585 learned_addr learned my addr 172.21.15.114:0/285666585 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5e159f700 1 -- 172.21.15.114:0/285666585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc005040 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5e159f700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc109330 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ff5cc000c00 tx=0x7ff5cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5cc0095f0 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff5cc007e50 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc060500 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5cc014e70 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.538+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc1099e0 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.539+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff5cc025070 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.539+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc0591e0 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.539+0000 7ff5d1ffb700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff5c80310e0 0x7ff5c80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.539+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff5cc041cd0 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.540+0000 7ff5e0d9e700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff5c80310e0 0x7ff5c80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.540+0000 7ff5e0d9e700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff5c80310e0 0x7ff5c80335a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff5d80096a0 tx=0x7ff5d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.545+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff5cc00f4e0 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:42.218187+0000 mgr.smithi038.powepi (mgr.14156) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:43.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.754+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5dc045bb0 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.754+0000 7ff5d1ffb700 1 -- 172.21.15.114:0/285666585 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5cc00f300 con 0x7ff5dc0f52d0 2024-02-20T13:25:43.756 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:43.756 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff5c80310e0 msgr2=0x7ff5c80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff5c80310e0 0x7ff5c80335a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff5d80096a0 tx=0x7ff5d8006b40).stop 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 msgr2=0x7ff5dc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc109330 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ff5cc000c00 tx=0x7ff5cc00b040).stop 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 shutdown_connections 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff5c80310e0 0x7ff5c80335a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 --2- 172.21.15.114:0/285666585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff5dc0f52d0 0x7ff5dc109330 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 >> 172.21.15.114:0/285666585 conn(0x7ff5dc0f0a30 msgr2=0x7ff5dc0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:43.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 shutdown_connections 2024-02-20T13:25:43.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:43.756+0000 7ff5e25a1700 1 -- 172.21.15.114:0/285666585 wait complete. 2024-02-20T13:25:43.759 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:44.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:43.756018+0000 mon.smithi038 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.114:0/285666585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:45.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:45.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:45.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:44.218431+0000 mgr.smithi038.powepi (mgr.14156) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 -- 172.21.15.114:0/3041075012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 msgr2=0x7fb99c0f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 --2- 172.21.15.114:0/3041075012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c0f8fb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb984009fb0 tx=0x7fb984009b30).stop 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 -- 172.21.15.114:0/3041075012 shutdown_connections 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 --2- 172.21.15.114:0/3041075012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c0f8fb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 -- 172.21.15.114:0/3041075012 >> 172.21.15.114:0/3041075012 conn(0x7fb99c0f0a30 msgr2=0x7fb99c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 -- 172.21.15.114:0/3041075012 shutdown_connections 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.676+0000 7fb9a084f700 1 -- 172.21.15.114:0/3041075012 wait complete. 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb9a084f700 1 Processor -- start 2024-02-20T13:25:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb9a084f700 1 -- start start 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb9a084f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c1065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb9a084f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c106aa0 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb99b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c1065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb99b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c1065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:56238/0 (socket says 172.21.15.114:56238) 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.677+0000 7fb99b7fe700 1 -- 172.21.15.114:0/1563137474 learned_addr learned my addr 172.21.15.114:0/1563137474 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb99b7fe700 1 -- 172.21.15.114:0/1563137474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb984005040 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb99b7fe700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c1065b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb984009f80 tx=0x7fb98400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb984009320 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb984013680 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c106ca0 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb98401eda0 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.678+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c107130 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.679+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb98401e420 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.679+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c045bb0 con 0x7fb99c060360 2024-02-20T13:25:47.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.679+0000 7fb993fff700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb9880310e0 0x7fb9880335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.679+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb9840423e0 con 0x7fb99c060360 2024-02-20T13:25:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.679+0000 7fb99affd700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb9880310e0 0x7fb9880335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.680+0000 7fb99affd700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb9880310e0 0x7fb9880335a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb98c0096a0 tx=0x7fb98c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:47.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.684+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb98401e6b0 con 0x7fb99c060360 2024-02-20T13:25:47.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:46.218738+0000 mgr.smithi038.powepi (mgr.14156) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:47.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.896+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb99c107fe0 con 0x7fb99c060360 2024-02-20T13:25:47.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.896+0000 7fb993fff700 1 -- 172.21.15.114:0/1563137474 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb984004490 con 0x7fb99c060360 2024-02-20T13:25:47.897 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:47.897 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb9880310e0 msgr2=0x7fb9880335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb9880310e0 0x7fb9880335a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb98c0096a0 tx=0x7fb98c006b40).stop 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 msgr2=0x7fb99c1065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c1065b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb984009f80 tx=0x7fb98400b040).stop 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 shutdown_connections 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb9880310e0 0x7fb9880335a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 --2- 172.21.15.114:0/1563137474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb99c060360 0x7fb99c1065b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:47.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 >> 172.21.15.114:0/1563137474 conn(0x7fb99c0f0a30 msgr2=0x7fb99c0f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:47.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 shutdown_connections 2024-02-20T13:25:47.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:47.898+0000 7fb9a084f700 1 -- 172.21.15.114:0/1563137474 wait complete. 2024-02-20T13:25:47.900 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:48.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:47.897191+0000 mon.smithi038 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.114:0/1563137474' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:49.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:49.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:48.218983+0000 mgr.smithi038.powepi (mgr.14156) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 -- 172.21.15.114:0/323698466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 msgr2=0x7f43a40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 --2- 172.21.15.114:0/323698466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a40f58a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4394009fb0 tx=0x7f4394009b30).stop 2024-02-20T13:25:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 -- 172.21.15.114:0/323698466 shutdown_connections 2024-02-20T13:25:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 --2- 172.21.15.114:0/323698466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a40f58a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 -- 172.21.15.114:0/323698466 >> 172.21.15.114:0/323698466 conn(0x7f43a40f0be0 msgr2=0x7f43a40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 -- 172.21.15.114:0/323698466 shutdown_connections 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.596+0000 7f43ac112700 1 -- 172.21.15.114:0/323698466 wait complete. 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43ac112700 1 Processor -- start 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43ac112700 1 -- start start 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43ac112700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a41056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43ac112700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4105be0 con 0x7f43a40f5480 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43a9eae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a41056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43a9eae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a41056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:56256/0 (socket says 172.21.15.114:56256) 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.597+0000 7f43a9eae700 1 -- 172.21.15.114:0/4166897141 learned_addr learned my addr 172.21.15.114:0/4166897141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f43a9eae700 1 -- 172.21.15.114:0/4166897141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394005040 con 0x7f43a40f5480 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f43a9eae700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a41056f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f43940089f0 tx=0x7f439400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4394009320 con 0x7f43a40f5480 2024-02-20T13:25:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a4105de0 con 0x7f43a40f5480 2024-02-20T13:25:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4394013ec0 con 0x7f43a40f5480 2024-02-20T13:25:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4394014d30 con 0x7f43a40f5480 2024-02-20T13:25:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.598+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a4106270 con 0x7f43a40f5480 2024-02-20T13:25:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.599+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4394014420 con 0x7f43a40f5480 2024-02-20T13:25:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.599+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a4045bb0 con 0x7f43a40f5480 2024-02-20T13:25:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.599+0000 7f439a7fc700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4390031090 0x7f4390033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.599+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4394042460 con 0x7f43a40f5480 2024-02-20T13:25:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.599+0000 7f43a96ad700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4390031090 0x7f4390033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.600+0000 7f43a96ad700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4390031090 0x7f4390033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f43a00096a0 tx=0x7f43a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.604+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f439401d7f0 con 0x7f43a40f5480 2024-02-20T13:25:51.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:50.219253+0000 mgr.smithi038.powepi (mgr.14156) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:51.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.809+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43a40faf70 con 0x7f43a40f5480 2024-02-20T13:25:51.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.810+0000 7f439a7fc700 1 -- 172.21.15.114:0/4166897141 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f439401d7f0 con 0x7f43a40f5480 2024-02-20T13:25:51.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:51.811 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:51.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.811+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4390031090 msgr2=0x7f4390033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:51.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.811+0000 7f43ac112700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4390031090 0x7f4390033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f43a00096a0 tx=0x7f43a0006b40).stop 2024-02-20T13:25:51.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.811+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 msgr2=0x7f43a41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:51.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a41056f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f43940089f0 tx=0x7f439400b040).stop 2024-02-20T13:25:51.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 shutdown_connections 2024-02-20T13:25:51.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4390031090 0x7f4390033550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:51.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 --2- 172.21.15.114:0/4166897141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43a40f5480 0x7f43a41056f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:51.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 >> 172.21.15.114:0/4166897141 conn(0x7f43a40f0be0 msgr2=0x7f43a40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:51.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 shutdown_connections 2024-02-20T13:25:51.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:51.812+0000 7f43ac112700 1 -- 172.21.15.114:0/4166897141 wait complete. 2024-02-20T13:25:51.813 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:52.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:51.810800+0000 mon.smithi038 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.114:0/4166897141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:53.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:52.219707+0000 mgr.smithi038.powepi (mgr.14156) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:53.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:53.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:55.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:54.220219+0000 mgr.smithi038.powepi (mgr.14156) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:55.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 -- 172.21.15.114:0/1377270459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 msgr2=0x7f1c240f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:55.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 --2- 172.21.15.114:0/1377270459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c240f58d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f1c14009fb0 tx=0x7f1c14009b30).stop 2024-02-20T13:25:55.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 -- 172.21.15.114:0/1377270459 shutdown_connections 2024-02-20T13:25:55.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 --2- 172.21.15.114:0/1377270459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c240f58d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:55.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 -- 172.21.15.114:0/1377270459 >> 172.21.15.114:0/1377270459 conn(0x7f1c240f0c10 msgr2=0x7f1c240f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:55.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 -- 172.21.15.114:0/1377270459 shutdown_connections 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.825+0000 7f1c2b30b700 1 -- 172.21.15.114:0/1377270459 wait complete. 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c2b30b700 1 Processor -- start 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c2b30b700 1 -- start start 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c2b30b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c241094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c2b30b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24060300 con 0x7f1c240f54b0 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c290a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c241094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c290a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c241094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41854/0 (socket says 172.21.15.114:41854) 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.826+0000 7f1c290a7700 1 -- 172.21.15.114:0/3410883642 learned_addr learned my addr 172.21.15.114:0/3410883642 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c290a7700 1 -- 172.21.15.114:0/3410883642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14005040 con 0x7f1c240f54b0 2024-02-20T13:25:55.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c290a7700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c241094c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f1c14000c00 tx=0x7f1c1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c140095f0 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1c14007e50 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c24060500 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14014e70 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.827+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c24109b70 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.828+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1c14025070 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.828+0000 7f1c19ffb700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c100310e0 0x7f1c100335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.828+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1c14041e10 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.828+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c240591e0 con 0x7f1c240f54b0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.828+0000 7f1c288a6700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c100310e0 0x7f1c100335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:55.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.829+0000 7f1c288a6700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c100310e0 0x7f1c100335a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1c200096a0 tx=0x7f1c20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:55.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:55.833+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1c14026350 con 0x7f1c240f54b0 2024-02-20T13:25:56.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.042+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c24045bb0 con 0x7f1c240f54b0 2024-02-20T13:25:56.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.043+0000 7f1c19ffb700 1 -- 172.21.15.114:0/3410883642 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c14012330 con 0x7f1c240f54b0 2024-02-20T13:25:56.044 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:56.044 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:56.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c100310e0 msgr2=0x7f1c100335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c100310e0 0x7f1c100335a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1c200096a0 tx=0x7f1c20006b40).stop 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 msgr2=0x7f1c241094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c241094c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f1c14000c00 tx=0x7f1c1400b040).stop 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 shutdown_connections 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c100310e0 0x7f1c100335a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 --2- 172.21.15.114:0/3410883642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c240f54b0 0x7f1c241094c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 >> 172.21.15.114:0/3410883642 conn(0x7f1c240f0c10 msgr2=0x7f1c240f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 shutdown_connections 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:56.045+0000 7f1c2b30b700 1 -- 172.21.15.114:0/3410883642 wait complete. 2024-02-20T13:25:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:25:56.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:56.043790+0000 mon.smithi038 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.114:0/3410883642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:25:57.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:25:57.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:25:57.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:56.220640+0000 mgr.smithi038.powepi (mgr.14156) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:59.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.595+0000 7fea8626f700 1 -- 172.21.15.114:0/3145420856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 msgr2=0x7fea800f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.595+0000 7fea8626f700 1 --2- 172.21.15.114:0/3145420856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea800f69b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fea70004760 tx=0x7fea70009ad0).stop 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.595+0000 7fea8626f700 1 -- 172.21.15.114:0/3145420856 shutdown_connections 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.595+0000 7fea8626f700 1 --2- 172.21.15.114:0/3145420856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea800f69b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.595+0000 7fea8626f700 1 -- 172.21.15.114:0/3145420856 >> 172.21.15.114:0/3145420856 conn(0x7fea800f1cd0 msgr2=0x7fea800f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.596+0000 7fea8626f700 1 -- 172.21.15.114:0/3145420856 shutdown_connections 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.596+0000 7fea8626f700 1 -- 172.21.15.114:0/3145420856 wait complete. 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.596+0000 7fea8626f700 1 Processor -- start 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.596+0000 7fea8626f700 1 -- start start 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.596+0000 7fea8626f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea801067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.596+0000 7fea8626f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80106cb0 con 0x7fea800f6590 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea801067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:59.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea801067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41862/0 (socket says 172.21.15.114:41862) 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7ffff700 1 -- 172.21.15.114:0/4046301713 learned_addr learned my addr 172.21.15.114:0/4046301713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7ffff700 1 -- 172.21.15.114:0/4046301713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea70009d70 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7ffff700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea801067c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fea70004170 tx=0x7fea70009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea7000e040 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea80106eb0 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fea70008ef0 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea70014ca0 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.597+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea80107340 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.598+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fea70014420 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.598+0000 7fea7cff9700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fea6c031090 0x7fea6c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.598+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea80045bb0 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.598+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fea700421d0 con 0x7fea800f6590 2024-02-20T13:25:59.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.598+0000 7fea7f7fe700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fea6c031090 0x7fea6c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:25:59.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.599+0000 7fea7f7fe700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fea6c031090 0x7fea6c033550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fea680096a0 tx=0x7fea68006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:25:59.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.603+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fea70014e00 con 0x7fea800f6590 2024-02-20T13:25:59.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:25:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:25:58.221078+0000 mgr.smithi038.powepi (mgr.14156) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:25:59.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.810+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea800fc170 con 0x7fea800f6590 2024-02-20T13:25:59.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.811+0000 7fea7cff9700 1 -- 172.21.15.114:0/4046301713 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea7001e7f0 con 0x7fea800f6590 2024-02-20T13:25:59.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:25:59.811 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fea6c031090 msgr2=0x7fea6c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fea6c031090 0x7fea6c033550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fea680096a0 tx=0x7fea68006b40).stop 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 msgr2=0x7fea801067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea801067c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fea70004170 tx=0x7fea70009ad0).stop 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 shutdown_connections 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fea6c031090 0x7fea6c033550 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 --2- 172.21.15.114:0/4046301713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800f6590 0x7fea801067c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 >> 172.21.15.114:0/4046301713 conn(0x7fea800f1cd0 msgr2=0x7fea800f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:25:59.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 shutdown_connections 2024-02-20T13:25:59.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:25:59.812+0000 7fea8626f700 1 -- 172.21.15.114:0/4046301713 wait complete. 2024-02-20T13:25:59.814 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:00.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:25:59.811367+0000 mon.smithi038 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.114:0/4046301713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:01.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:01.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:00.221424+0000 mgr.smithi038.powepi (mgr.14156) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 -- 172.21.15.114:0/1523860661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 msgr2=0x7f6b6c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 --2- 172.21.15.114:0/1523860661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c0f58d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f6b5c009fb0 tx=0x7f6b5c009b30).stop 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 -- 172.21.15.114:0/1523860661 shutdown_connections 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 --2- 172.21.15.114:0/1523860661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c0f58d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 -- 172.21.15.114:0/1523860661 >> 172.21.15.114:0/1523860661 conn(0x7f6b6c0f0c10 msgr2=0x7f6b6c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 -- 172.21.15.114:0/1523860661 shutdown_connections 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.629+0000 7f6b71309700 1 -- 172.21.15.114:0/1523860661 wait complete. 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b71309700 1 Processor -- start 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b71309700 1 -- start start 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b71309700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b71309700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c106ca0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b6affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b6affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41882/0 (socket says 172.21.15.114:41882) 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b6affd700 1 -- 172.21.15.114:0/336654132 learned_addr learned my addr 172.21.15.114:0/336654132 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b6affd700 1 -- 172.21.15.114:0/336654132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c005040 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b6affd700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c1067b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f6b5c0089f0 tx=0x7f6b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.630+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b5c009320 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.631+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c106ea0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.631+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6b5c013ec0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.631+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b5c014d30 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.631+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c107330 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.631+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6b5c0095f0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.632+0000 7f6b537fe700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6b54031090 0x7f6b54033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.632+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c045bb0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.632+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6b5c0423e0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.632+0000 7f6b6a7fc700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6b54031090 0x7f6b54033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.632+0000 7f6b6a7fc700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6b54031090 0x7f6b54033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f6b600096a0 tx=0x7f6b60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.637+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6b5c01d7f0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:02.221769+0000 mgr.smithi038.powepi (mgr.14156) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:03.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.844+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b6c1081b0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.844+0000 7f6b537fe700 1 -- 172.21.15.114:0/336654132 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b5c01d7f0 con 0x7f6b6c0f54b0 2024-02-20T13:26:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:03.845 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6b54031090 msgr2=0x7f6b54033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6b54031090 0x7f6b54033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f6b600096a0 tx=0x7f6b60006b40).stop 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 msgr2=0x7f6b6c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c1067b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f6b5c0089f0 tx=0x7f6b5c00b040).stop 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 shutdown_connections 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6b54031090 0x7f6b54033550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 --2- 172.21.15.114:0/336654132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b6c0f54b0 0x7f6b6c1067b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 >> 172.21.15.114:0/336654132 conn(0x7f6b6c0f0c10 msgr2=0x7f6b6c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 shutdown_connections 2024-02-20T13:26:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:03.846+0000 7f6b71309700 1 -- 172.21.15.114:0/336654132 wait complete. 2024-02-20T13:26:03.848 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:04.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:03.844956+0000 mon.smithi038 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.114:0/336654132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:05.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:05.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:05.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:04.222002+0000 mgr.smithi038.powepi (mgr.14156) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- 172.21.15.114:0/492088164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 msgr2=0x7f994c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 --2- 172.21.15.114:0/492088164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c0f56f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f993c009fb0 tx=0x7f993c009b30).stop 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- 172.21.15.114:0/492088164 shutdown_connections 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 --2- 172.21.15.114:0/492088164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c0f56f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- 172.21.15.114:0/492088164 >> 172.21.15.114:0/492088164 conn(0x7f994c0f0a30 msgr2=0x7f994c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- 172.21.15.114:0/492088164 shutdown_connections 2024-02-20T13:26:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- 172.21.15.114:0/492088164 wait complete. 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 Processor -- start 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- start start 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.683+0000 7f994b59e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c060320 con 0x7f994c0f52d0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47986/0 (socket says 172.21.15.114:47986) 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994a59c700 1 -- 172.21.15.114:0/4095540057 learned_addr learned my addr 172.21.15.114:0/4095540057 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994a59c700 1 -- 172.21.15.114:0/4095540057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c005040 con 0x7f994c0f52d0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994a59c700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c109310 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f993c0089f0 tx=0x7f993c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993c009320 con 0x7f994c0f52d0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c060520 con 0x7f994c0f52d0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f993c013ec0 con 0x7f994c0f52d0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993c014d30 con 0x7f994c0f52d0 2024-02-20T13:26:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.684+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c1099c0 con 0x7f994c0f52d0 2024-02-20T13:26:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.685+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f993c014420 con 0x7f994c0f52d0 2024-02-20T13:26:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.685+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c045bb0 con 0x7f994c0f52d0 2024-02-20T13:26:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.685+0000 7f993affd700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9934031090 0x7f9934033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.685+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f993c042400 con 0x7f994c0f52d0 2024-02-20T13:26:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.685+0000 7f9949d9b700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9934031090 0x7f9934033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.686+0000 7f9949d9b700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9934031090 0x7f9934033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f99400096a0 tx=0x7f9940006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.690+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f993c01d980 con 0x7f994c0f52d0 2024-02-20T13:26:07.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:06.222370+0000 mgr.smithi038.powepi (mgr.14156) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:07.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.898+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f994c10a1e0 con 0x7f994c0f52d0 2024-02-20T13:26:07.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.899+0000 7f993affd700 1 -- 172.21.15.114:0/4095540057 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f993c01d380 con 0x7f994c0f52d0 2024-02-20T13:26:07.900 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:07.900 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9934031090 msgr2=0x7f9934033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9934031090 0x7f9934033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f99400096a0 tx=0x7f9940006b40).stop 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 msgr2=0x7f994c109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c109310 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f993c0089f0 tx=0x7f993c00b040).stop 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 shutdown_connections 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9934031090 0x7f9934033550 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 --2- 172.21.15.114:0/4095540057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f994c0f52d0 0x7f994c109310 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 >> 172.21.15.114:0/4095540057 conn(0x7f994c0f0a30 msgr2=0x7f994c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 shutdown_connections 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:07.901+0000 7f994b59e700 1 -- 172.21.15.114:0/4095540057 wait complete. 2024-02-20T13:26:07.902 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:08.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:07.899416+0000 mon.smithi038 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.114:0/4095540057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:09.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:09.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:08.222686+0000 mgr.smithi038.powepi (mgr.14156) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 -- 172.21.15.114:0/1639847423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 msgr2=0x7f46380f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 --2- 172.21.15.114:0/1639847423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f46380f58a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f462c009fb0 tx=0x7f462c009b30).stop 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 -- 172.21.15.114:0/1639847423 shutdown_connections 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 --2- 172.21.15.114:0/1639847423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f46380f58a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 -- 172.21.15.114:0/1639847423 >> 172.21.15.114:0/1639847423 conn(0x7f46380f0be0 msgr2=0x7f46380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 -- 172.21.15.114:0/1639847423 shutdown_connections 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.481+0000 7f463d492700 1 -- 172.21.15.114:0/1639847423 wait complete. 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f463d492700 1 Processor -- start 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f463d492700 1 -- start start 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f463d492700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f463810b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f463d492700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4638107300 con 0x7f46380f5480 2024-02-20T13:26:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f4636ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f463810b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f4636ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f463810b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:48004/0 (socket says 172.21.15.114:48004) 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f4636ffd700 1 -- 172.21.15.114:0/1874706006 learned_addr learned my addr 172.21.15.114:0/1874706006 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f4636ffd700 1 -- 172.21.15.114:0/1874706006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c005040 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.482+0000 7f4636ffd700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f463810b910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f462c0089f0 tx=0x7f462c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.483+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f462c009320 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.483+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4638060300 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.483+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f462c013ec0 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.483+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f462c014d30 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.483+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463810be00 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.484+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f462c0095f0 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.484+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4638045bb0 con 0x7f46380f5480 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.484+0000 7f461f7fe700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4620031090 0x7f4620033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.484+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f462c0423e0 con 0x7f46380f5480 2024-02-20T13:26:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.484+0000 7f46367fc700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4620031090 0x7f4620033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.484+0000 7f46367fc700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4620031090 0x7f4620033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f46280096a0 tx=0x7f4628006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.488+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f462c01d7f0 con 0x7f46380f5480 2024-02-20T13:26:11.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:10.223074+0000 mgr.smithi038.powepi (mgr.14156) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:11.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.696+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46380faf70 con 0x7f46380f5480 2024-02-20T13:26:11.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.697+0000 7f461f7fe700 1 -- 172.21.15.114:0/1874706006 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f462c01d7f0 con 0x7f46380f5480 2024-02-20T13:26:11.699 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:11.699 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:11.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4620031090 msgr2=0x7f4620033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:11.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4620031090 0x7f4620033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f46280096a0 tx=0x7f4628006b40).stop 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 msgr2=0x7f463810b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f463810b910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f462c0089f0 tx=0x7f462c00b040).stop 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 shutdown_connections 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f4620031090 0x7f4620033550 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.698+0000 7f463d492700 1 --2- 172.21.15.114:0/1874706006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46380f5480 0x7f463810b910 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.699+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 >> 172.21.15.114:0/1874706006 conn(0x7f46380f0be0 msgr2=0x7f46380f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.699+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 shutdown_connections 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:11.699+0000 7f463d492700 1 -- 172.21.15.114:0/1874706006 wait complete. 2024-02-20T13:26:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:12.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:11.698864+0000 mon.smithi038 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.114:0/1874706006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:13.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:13.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:13.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:12.223394+0000 mgr.smithi038.powepi (mgr.14156) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:15.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.548+0000 7f6608b7a700 1 -- 172.21.15.114:0/1928833037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 msgr2=0x7f66040f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:15.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.548+0000 7f6608b7a700 1 --2- 172.21.15.114:0/1928833037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f66040f69b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f65ec004760 tx=0x7f65ec009ad0).stop 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.548+0000 7f6608b7a700 1 -- 172.21.15.114:0/1928833037 shutdown_connections 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.548+0000 7f6608b7a700 1 --2- 172.21.15.114:0/1928833037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f66040f69b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.548+0000 7f6608b7a700 1 -- 172.21.15.114:0/1928833037 >> 172.21.15.114:0/1928833037 conn(0x7f66040f1cd0 msgr2=0x7f66040f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f6608b7a700 1 -- 172.21.15.114:0/1928833037 shutdown_connections 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f6608b7a700 1 -- 172.21.15.114:0/1928833037 wait complete. 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f6608b7a700 1 Processor -- start 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f6608b7a700 1 -- start start 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f6608b7a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f6604108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f6608b7a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6604108f00 con 0x7f66040f6590 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f660259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f6604108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f660259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f6604108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55872/0 (socket says 172.21.15.114:55872) 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.549+0000 7f660259c700 1 -- 172.21.15.114:0/1875606819 learned_addr learned my addr 172.21.15.114:0/1875606819 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f660259c700 1 -- 172.21.15.114:0/1875606819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec009d70 con 0x7f66040f6590 2024-02-20T13:26:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f660259c700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f6604108a10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f65ec004170 tx=0x7f65ec009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec00e040 con 0x7f66040f6590 2024-02-20T13:26:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6604109100 con 0x7f66040f6590 2024-02-20T13:26:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f65ec008ef0 con 0x7f66040f6590 2024-02-20T13:26:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec014c90 con 0x7f66040f6590 2024-02-20T13:26:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.550+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6604109590 con 0x7f66040f6590 2024-02-20T13:26:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.551+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f65ec0283f0 con 0x7f66040f6590 2024-02-20T13:26:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.551+0000 7f65faffd700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f65f0031090 0x7f65f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.551+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f65ec042260 con 0x7f66040f6590 2024-02-20T13:26:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.551+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66040591e0 con 0x7f66040f6590 2024-02-20T13:26:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.551+0000 7f6601d9b700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f65f0031090 0x7f65f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.552+0000 7f6601d9b700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f65f0031090 0x7f65f0033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f65f40096a0 tx=0x7f65f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:15.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.556+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f65ec0268f0 con 0x7f66040f6590 2024-02-20T13:26:15.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:14.223759+0000 mgr.smithi038.powepi (mgr.14156) 274 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:15.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.765+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6604045bb0 con 0x7f66040f6590 2024-02-20T13:26:15.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.766+0000 7f65faffd700 1 -- 172.21.15.114:0/1875606819 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65ec026300 con 0x7f66040f6590 2024-02-20T13:26:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:15.768 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f65f0031090 msgr2=0x7f65f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f65f0031090 0x7f65f0033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f65f40096a0 tx=0x7f65f4006b40).stop 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 msgr2=0x7f6604108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f6604108a10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f65ec004170 tx=0x7f65ec009ad0).stop 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 shutdown_connections 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f65f0031090 0x7f65f0033550 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 --2- 172.21.15.114:0/1875606819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66040f6590 0x7f6604108a10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 >> 172.21.15.114:0/1875606819 conn(0x7f66040f1cd0 msgr2=0x7f66040f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 shutdown_connections 2024-02-20T13:26:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:15.768+0000 7f6608b7a700 1 -- 172.21.15.114:0/1875606819 wait complete. 2024-02-20T13:26:15.771 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:16.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:15.767877+0000 mon.smithi038 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.114:0/1875606819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:17.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:17.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:17.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:16.224133+0000 mgr.smithi038.powepi (mgr.14156) 275 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:19.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- 172.21.15.114:0/2773704396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 msgr2=0x7fa5040f4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:19.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 --2- 172.21.15.114:0/2773704396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa5040f4ad0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fa4f4009fb0 tx=0x7fa4f4009b30).stop 2024-02-20T13:26:19.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- 172.21.15.114:0/2773704396 shutdown_connections 2024-02-20T13:26:19.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 --2- 172.21.15.114:0/2773704396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa5040f4ad0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- 172.21.15.114:0/2773704396 >> 172.21.15.114:0/2773704396 conn(0x7fa5040f01d0 msgr2=0x7fa5040f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- 172.21.15.114:0/2773704396 shutdown_connections 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- 172.21.15.114:0/2773704396 wait complete. 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 Processor -- start 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- start start 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa504108820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.367+0000 7fa509346700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa504108d10 con 0x7fa5040f46b0 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa503fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa504108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa503fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa504108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55900/0 (socket says 172.21.15.114:55900) 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa503fff700 1 -- 172.21.15.114:0/3705566304 learned_addr learned my addr 172.21.15.114:0/3705566304 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa503fff700 1 -- 172.21.15.114:0/3705566304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f4005040 con 0x7fa5040f46b0 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa503fff700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa504108820 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fa4f40089f0 tx=0x7fa4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4f4009320 con 0x7fa5040f46b0 2024-02-20T13:26:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa4f4013ec0 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa504108f10 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4f4014d30 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.368+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5041093a0 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.369+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa4f40095f0 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.369+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa504045bb0 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.369+0000 7fa500ff9700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa4ec031090 0x7fa4ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.369+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa4f40424a0 con 0x7fa5040f46b0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.370+0000 7fa5037fe700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa4ec031090 0x7fa4ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:19.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.370+0000 7fa5037fe700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa4ec031090 0x7fa4ec033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa4f80096a0 tx=0x7fa4f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:19.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.374+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa4f40268c0 con 0x7fa5040f46b0 2024-02-20T13:26:19.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.579+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5040f6aa0 con 0x7fa5040f46b0 2024-02-20T13:26:19.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.580+0000 7fa500ff9700 1 -- 172.21.15.114:0/3705566304 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4f401d7f0 con 0x7fa5040f46b0 2024-02-20T13:26:19.581 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:19.582 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa4ec031090 msgr2=0x7fa4ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa4ec031090 0x7fa4ec033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa4f80096a0 tx=0x7fa4f8006b40).stop 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 msgr2=0x7fa504108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa504108820 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fa4f40089f0 tx=0x7fa4f400b040).stop 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 shutdown_connections 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa4ec031090 0x7fa4ec033550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 --2- 172.21.15.114:0/3705566304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5040f46b0 0x7fa504108820 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 >> 172.21.15.114:0/3705566304 conn(0x7fa5040f01d0 msgr2=0x7fa5040f0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 shutdown_connections 2024-02-20T13:26:19.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:19.581+0000 7fa509346700 1 -- 172.21.15.114:0/3705566304 wait complete. 2024-02-20T13:26:19.584 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:19.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:18.224395+0000 mgr.smithi038.powepi (mgr.14156) 276 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:20.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:19.581345+0000 mon.smithi038 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.114:0/3705566304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:21.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:21.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:21.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:20.224658+0000 mgr.smithi038.powepi (mgr.14156) 277 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 -- 172.21.15.114:0/2985657927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 msgr2=0x7f9b500f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 --2- 172.21.15.114:0/2985657927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b500f4c70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f9b44009fb0 tx=0x7f9b44009b30).stop 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 -- 172.21.15.114:0/2985657927 shutdown_connections 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 --2- 172.21.15.114:0/2985657927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b500f4c70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 -- 172.21.15.114:0/2985657927 >> 172.21.15.114:0/2985657927 conn(0x7f9b500f0390 msgr2=0x7f9b500f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 -- 172.21.15.114:0/2985657927 shutdown_connections 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.600+0000 7f9b56f84700 1 -- 172.21.15.114:0/2985657927 wait complete. 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b56f84700 1 Processor -- start 2024-02-20T13:26:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b56f84700 1 -- start start 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b56f84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b501067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b56f84700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b50106c90 con 0x7f9b500f4850 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b54d20700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b501067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b54d20700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b501067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55910/0 (socket says 172.21.15.114:55910) 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.601+0000 7f9b54d20700 1 -- 172.21.15.114:0/217450336 learned_addr learned my addr 172.21.15.114:0/217450336 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b54d20700 1 -- 172.21.15.114:0/217450336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44005040 con 0x7f9b500f4850 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b54d20700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b501067a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f9b440089f0 tx=0x7f9b4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b44009320 con 0x7f9b500f4850 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b50106e90 con 0x7f9b500f4850 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9b44013ec0 con 0x7f9b500f4850 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b44014d30 con 0x7f9b500f4850 2024-02-20T13:26:23.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.602+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b50107320 con 0x7f9b500f4850 2024-02-20T13:26:23.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.603+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9b440095f0 con 0x7f9b500f4850 2024-02-20T13:26:23.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.603+0000 7f9b4d7fa700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b40031090 0x7f9b40033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:23.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.603+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9b44042440 con 0x7f9b500f4850 2024-02-20T13:26:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.603+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b50045bb0 con 0x7f9b500f4850 2024-02-20T13:26:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.603+0000 7f9b4ffff700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b40031090 0x7f9b40033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.604+0000 7f9b4ffff700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b40031090 0x7f9b40033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9b3c0096a0 tx=0x7f9b3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:23.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.608+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9b4401dbe0 con 0x7f9b500f4850 2024-02-20T13:26:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:22.224943+0000 mgr.smithi038.powepi (mgr.14156) 278 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:23.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.813+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b500fc170 con 0x7f9b500f4850 2024-02-20T13:26:23.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.814+0000 7f9b4d7fa700 1 -- 172.21.15.114:0/217450336 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b4402b020 con 0x7f9b500f4850 2024-02-20T13:26:23.816 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:23.816 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b40031090 msgr2=0x7f9b40033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b40031090 0x7f9b40033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9b3c0096a0 tx=0x7f9b3c006b40).stop 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 msgr2=0x7f9b501067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b501067a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f9b440089f0 tx=0x7f9b4400b040).stop 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 shutdown_connections 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9b40031090 0x7f9b40033550 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 --2- 172.21.15.114:0/217450336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b500f4850 0x7f9b501067a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:23.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 >> 172.21.15.114:0/217450336 conn(0x7f9b500f0390 msgr2=0x7f9b500f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:23.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 shutdown_connections 2024-02-20T13:26:23.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:23.815+0000 7f9b56f84700 1 -- 172.21.15.114:0/217450336 wait complete. 2024-02-20T13:26:23.818 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:24.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:23.815563+0000 mon.smithi038 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.114:0/217450336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:25.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:25.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:24.225306+0000 mgr.smithi038.powepi (mgr.14156) 279 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:27.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 -- 172.21.15.114:0/3667357434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 msgr2=0x7fdb680f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:27.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 --2- 172.21.15.114:0/3667357434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f9200 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fdb58009fb0 tx=0x7fdb58009b30).stop 2024-02-20T13:26:27.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 -- 172.21.15.114:0/3667357434 shutdown_connections 2024-02-20T13:26:27.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 --2- 172.21.15.114:0/3667357434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f9200 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:27.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 -- 172.21.15.114:0/3667357434 >> 172.21.15.114:0/3667357434 conn(0x7fdb680f0be0 msgr2=0x7fdb680f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:27.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 -- 172.21.15.114:0/3667357434 shutdown_connections 2024-02-20T13:26:27.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.500+0000 7fdb705eb700 1 -- 172.21.15.114:0/3667357434 wait complete. 2024-02-20T13:26:27.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb705eb700 1 Processor -- start 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb705eb700 1 -- start start 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb705eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb705eb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb680f5100 con 0x7fdb680602c0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb6e387700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb6e387700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43374/0 (socket says 172.21.15.114:43374) 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.501+0000 7fdb6e387700 1 -- 172.21.15.114:0/547832405 learned_addr learned my addr 172.21.15.114:0/547832405 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb6e387700 1 -- 172.21.15.114:0/547832405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58005040 con 0x7fdb680602c0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb6e387700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f6c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fdb58013fd0 tx=0x7fdb58009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb58018070 con 0x7fdb680602c0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb680f52a0 con 0x7fdb680602c0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdb58007b10 con 0x7fdb680602c0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb5801c3c0 con 0x7fdb680602c0 2024-02-20T13:26:27.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.502+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb680f5730 con 0x7fdb680602c0 2024-02-20T13:26:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.503+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdb58007c80 con 0x7fdb680602c0 2024-02-20T13:26:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.503+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb680591e0 con 0x7fdb680602c0 2024-02-20T13:26:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.503+0000 7fdb5effd700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb54031090 0x7fdb54033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.503+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdb58042a50 con 0x7fdb680602c0 2024-02-20T13:26:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.503+0000 7fdb6db86700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb54031090 0x7fdb54033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.504+0000 7fdb6db86700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb54031090 0x7fdb54033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdb640096a0 tx=0x7fdb64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:27.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.508+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdb58025b40 con 0x7fdb680602c0 2024-02-20T13:26:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:26.225662+0000 mgr.smithi038.powepi (mgr.14156) 280 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:27.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.714+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb680fc070 con 0x7fdb680602c0 2024-02-20T13:26:27.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.715+0000 7fdb5effd700 1 -- 172.21.15.114:0/547832405 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb58025b40 con 0x7fdb680602c0 2024-02-20T13:26:27.716 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:27.717 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:27.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb54031090 msgr2=0x7fdb54033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:27.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb54031090 0x7fdb54033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdb640096a0 tx=0x7fdb64006b40).stop 2024-02-20T13:26:27.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 msgr2=0x7fdb680f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:27.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f6c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fdb58013fd0 tx=0x7fdb58009b30).stop 2024-02-20T13:26:27.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 shutdown_connections 2024-02-20T13:26:27.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fdb54031090 0x7fdb54033550 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:27.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 --2- 172.21.15.114:0/547832405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb680602c0 0x7fdb680f6c80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:27.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 >> 172.21.15.114:0/547832405 conn(0x7fdb680f0be0 msgr2=0x7fdb680f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:27.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 shutdown_connections 2024-02-20T13:26:27.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:27.716+0000 7fdb705eb700 1 -- 172.21.15.114:0/547832405 wait complete. 2024-02-20T13:26:27.719 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:28.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:27.716472+0000 mon.smithi038 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.114:0/547832405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:29.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:29.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:28.225986+0000 mgr.smithi038.powepi (mgr.14156) 281 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:28.283945+0000 mon.smithi038 (mon.0) 268 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:26:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:28.284175+0000 mon.smithi038 (mon.0) 269 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 -- 172.21.15.114:0/1778933893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 msgr2=0x7f69f40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 --2- 172.21.15.114:0/1778933893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f40f56f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f69f0009fb0 tx=0x7f69f0009b30).stop 2024-02-20T13:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 -- 172.21.15.114:0/1778933893 shutdown_connections 2024-02-20T13:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 --2- 172.21.15.114:0/1778933893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f40f56f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 -- 172.21.15.114:0/1778933893 >> 172.21.15.114:0/1778933893 conn(0x7f69f40f0a30 msgr2=0x7f69f40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 -- 172.21.15.114:0/1778933893 shutdown_connections 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 -- 172.21.15.114:0/1778933893 wait complete. 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 Processor -- start 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.657+0000 7f69fa5cf700 1 -- start start 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69fa5cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f4105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69fa5cf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4105a00 con 0x7f69f40f52d0 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69f95cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f4105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69f95cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f4105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43390/0 (socket says 172.21.15.114:43390) 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69f95cd700 1 -- 172.21.15.114:0/107051922 learned_addr learned my addr 172.21.15.114:0/107051922 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69f95cd700 1 -- 172.21.15.114:0/107051922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0005040 con 0x7f69f40f52d0 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.658+0000 7f69f95cd700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f4105510 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f69f00089f0 tx=0x7f69f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.659+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69f0009320 con 0x7f69f40f52d0 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.659+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f4105c00 con 0x7f69f40f52d0 2024-02-20T13:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.659+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f69f0013ec0 con 0x7f69f40f52d0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.659+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69f0014d30 con 0x7f69f40f52d0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.659+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f4106090 con 0x7f69f40f52d0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.660+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f69f00095f0 con 0x7f69f40f52d0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.660+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f4045bb0 con 0x7f69f40f52d0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.660+0000 7f69e9ffb700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f69e0031090 0x7f69e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.660+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f69f00423e0 con 0x7f69f40f52d0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.660+0000 7f69f8dcc700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f69e0031090 0x7f69e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.660+0000 7f69f8dcc700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f69e0031090 0x7f69e0033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f69e40096a0 tx=0x7f69e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.665+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f69f001d7f0 con 0x7f69f40f52d0 2024-02-20T13:26:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:30.226277+0000 mgr.smithi038.powepi (mgr.14156) 282 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:31.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.873+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69f4106f40 con 0x7f69f40f52d0 2024-02-20T13:26:31.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.874+0000 7f69e9ffb700 1 -- 172.21.15.114:0/107051922 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69f001d7f0 con 0x7f69f40f52d0 2024-02-20T13:26:31.876 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:31.876 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:31.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f69e0031090 msgr2=0x7f69e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:31.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f69e0031090 0x7f69e0033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f69e40096a0 tx=0x7f69e4006b40).stop 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 msgr2=0x7f69f4105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f4105510 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f69f00089f0 tx=0x7f69f000b040).stop 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 shutdown_connections 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f69e0031090 0x7f69e0033550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 --2- 172.21.15.114:0/107051922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f40f52d0 0x7f69f4105510 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 >> 172.21.15.114:0/107051922 conn(0x7f69f40f0a30 msgr2=0x7f69f40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 shutdown_connections 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:31.876+0000 7f69fa5cf700 1 -- 172.21.15.114:0/107051922 wait complete. 2024-02-20T13:26:31.878 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:32.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:31.875915+0000 mon.smithi038 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.114:0/107051922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:33.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:33.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:33.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:32.226712+0000 mgr.smithi038.powepi (mgr.14156) 283 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.553+0000 7febc777c700 1 -- 172.21.15.114:0/20412010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 msgr2=0x7febc00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 --2- 172.21.15.114:0/20412010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc00f58a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7febb4009fb0 tx=0x7febb4009b30).stop 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 -- 172.21.15.114:0/20412010 shutdown_connections 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 --2- 172.21.15.114:0/20412010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc00f58a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 -- 172.21.15.114:0/20412010 >> 172.21.15.114:0/20412010 conn(0x7febc00f0be0 msgr2=0x7febc00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 -- 172.21.15.114:0/20412010 shutdown_connections 2024-02-20T13:26:35.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 -- 172.21.15.114:0/20412010 wait complete. 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 Processor -- start 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 -- start start 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.554+0000 7febc777c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc0105be0 con 0x7febc00f5480 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc5518700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc5518700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:57278/0 (socket says 172.21.15.114:57278) 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc5518700 1 -- 172.21.15.114:0/3896015798 learned_addr learned my addr 172.21.15.114:0/3896015798 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc5518700 1 -- 172.21.15.114:0/3896015798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb4005040 con 0x7febc00f5480 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc5518700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc01056f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7febb4000c00 tx=0x7febb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febb40095f0 con 0x7febc00f5480 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc0105de0 con 0x7febc00f5480 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7febb4007e50 con 0x7febc00f5480 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febb4014e70 con 0x7febc00f5480 2024-02-20T13:26:35.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.555+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc0106270 con 0x7febc00f5480 2024-02-20T13:26:35.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.556+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7febb4025070 con 0x7febc00f5480 2024-02-20T13:26:35.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.556+0000 7febb1ffb700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7febac031090 0x7febac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.556+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7febb4041e30 con 0x7febc00f5480 2024-02-20T13:26:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.556+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc00591e0 con 0x7febc00f5480 2024-02-20T13:26:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.557+0000 7febc4d17700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7febac031090 0x7febac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.557+0000 7febc4d17700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7febac031090 0x7febac033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7febbc0096a0 tx=0x7febbc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.561+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7febb4026350 con 0x7febc00f5480 2024-02-20T13:26:35.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:34.227074+0000 mgr.smithi038.powepi (mgr.14156) 284 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:35.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.768+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febc00faf70 con 0x7febc00f5480 2024-02-20T13:26:35.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.769+0000 7febb1ffb700 1 -- 172.21.15.114:0/3896015798 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febb402d090 con 0x7febc00f5480 2024-02-20T13:26:35.770 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:35.770 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7febac031090 msgr2=0x7febac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7febac031090 0x7febac033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7febbc0096a0 tx=0x7febbc006b40).stop 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 msgr2=0x7febc01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc01056f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7febb4000c00 tx=0x7febb400b040).stop 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 shutdown_connections 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7febac031090 0x7febac033550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 --2- 172.21.15.114:0/3896015798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febc00f5480 0x7febc01056f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 >> 172.21.15.114:0/3896015798 conn(0x7febc00f0be0 msgr2=0x7febc00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 shutdown_connections 2024-02-20T13:26:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:35.771+0000 7febc777c700 1 -- 172.21.15.114:0/3896015798 wait complete. 2024-02-20T13:26:35.773 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:36.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:35.769928+0000 mon.smithi038 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.114:0/3896015798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:37.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:37.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:37.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:36.227349+0000 mgr.smithi038.powepi (mgr.14156) 285 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 -- 172.21.15.114:0/2904581569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 msgr2=0x7f28440f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 --2- 172.21.15.114:0/2904581569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28440f58a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f282c009fb0 tx=0x7f282c009b30).stop 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 -- 172.21.15.114:0/2904581569 shutdown_connections 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 --2- 172.21.15.114:0/2904581569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28440f58a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 -- 172.21.15.114:0/2904581569 >> 172.21.15.114:0/2904581569 conn(0x7f28440f0be0 msgr2=0x7f28440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 -- 172.21.15.114:0/2904581569 shutdown_connections 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.461+0000 7f2849a71700 1 -- 172.21.15.114:0/2904581569 wait complete. 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f2849a71700 1 Processor -- start 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f2849a71700 1 -- start start 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f2849a71700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28441056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f2849a71700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2844105be0 con 0x7f28440f5480 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f28437fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28441056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f28437fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28441056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:57304/0 (socket says 172.21.15.114:57304) 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.462+0000 7f28437fe700 1 -- 172.21.15.114:0/421583215 learned_addr learned my addr 172.21.15.114:0/421583215 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f28437fe700 1 -- 172.21.15.114:0/421583215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c005040 con 0x7f28440f5480 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f28437fe700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28441056f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f282c0089f0 tx=0x7f282c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282c009320 con 0x7f28440f5480 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f282c013ec0 con 0x7f28440f5480 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2844105de0 con 0x7f28440f5480 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282c014d30 con 0x7f28440f5480 2024-02-20T13:26:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.463+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2844106270 con 0x7f28440f5480 2024-02-20T13:26:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.464+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f282c0095f0 con 0x7f28440f5480 2024-02-20T13:26:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.464+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2844045bb0 con 0x7f28440f5480 2024-02-20T13:26:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.464+0000 7f283bfff700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2830031090 0x7f2830033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.464+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f282c0441c0 con 0x7f28440f5480 2024-02-20T13:26:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.464+0000 7f2842ffd700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2830031090 0x7f2830033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.465+0000 7f2842ffd700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2830031090 0x7f2830033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f28340096a0 tx=0x7f2834006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:39.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.469+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f282c01f380 con 0x7f28440f5480 2024-02-20T13:26:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.673+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28440fb060 con 0x7f28440f5480 2024-02-20T13:26:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.674+0000 7f283bfff700 1 -- 172.21.15.114:0/421583215 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f282c0262b0 con 0x7f28440f5480 2024-02-20T13:26:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:39.675 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2830031090 msgr2=0x7f2830033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2830031090 0x7f2830033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f28340096a0 tx=0x7f2834006b40).stop 2024-02-20T13:26:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 msgr2=0x7f28441056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28441056f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f282c0089f0 tx=0x7f282c00b040).stop 2024-02-20T13:26:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 shutdown_connections 2024-02-20T13:26:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2830031090 0x7f2830033550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 --2- 172.21.15.114:0/421583215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28440f5480 0x7f28441056f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 >> 172.21.15.114:0/421583215 conn(0x7f28440f0be0 msgr2=0x7f28440f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 shutdown_connections 2024-02-20T13:26:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:39.675+0000 7f2849a71700 1 -- 172.21.15.114:0/421583215 wait complete. 2024-02-20T13:26:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:39.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:39 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:38.227640+0000 mgr.smithi038.powepi (mgr.14156) 286 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:40.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:39.674594+0000 mon.smithi038 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.114:0/421583215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:41.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:41.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:41.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:40.227995+0000 mgr.smithi038.powepi (mgr.14156) 287 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:43.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 -- 172.21.15.114:0/3578738524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 msgr2=0x7fced80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:43.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 --2- 172.21.15.114:0/3578738524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced80f58a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fcecc009fb0 tx=0x7fcecc009b30).stop 2024-02-20T13:26:43.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 -- 172.21.15.114:0/3578738524 shutdown_connections 2024-02-20T13:26:43.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 --2- 172.21.15.114:0/3578738524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced80f58a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:43.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 -- 172.21.15.114:0/3578738524 >> 172.21.15.114:0/3578738524 conn(0x7fced80f0be0 msgr2=0x7fced80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 -- 172.21.15.114:0/3578738524 shutdown_connections 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.180+0000 7fcede98f700 1 -- 172.21.15.114:0/3578738524 wait complete. 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fcede98f700 1 Processor -- start 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fcede98f700 1 -- start start 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fcede98f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced81056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fcede98f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced8105be0 con 0x7fced80f5480 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fced7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced81056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fced7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced81056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:57318/0 (socket says 172.21.15.114:57318) 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fced7fff700 1 -- 172.21.15.114:0/2602710864 learned_addr learned my addr 172.21.15.114:0/2602710864 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.181+0000 7fced7fff700 1 -- 172.21.15.114:0/2602710864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc005040 con 0x7fced80f5480 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.182+0000 7fced7fff700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced81056f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fcecc000c00 tx=0x7fcecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.182+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcecc0095f0 con 0x7fced80f5480 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.182+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced8105de0 con 0x7fced80f5480 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.182+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcecc007e50 con 0x7fced80f5480 2024-02-20T13:26:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.182+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcecc014e70 con 0x7fced80f5480 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.182+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced8106270 con 0x7fced80f5480 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.183+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcecc0146b0 con 0x7fced80f5480 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.183+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced8045bb0 con 0x7fced80f5480 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.183+0000 7fced4ff9700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcec0031090 0x7fcec0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.183+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcecc041c50 con 0x7fced80f5480 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.183+0000 7fced77fe700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcec0031090 0x7fcec0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.183+0000 7fced77fe700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcec0031090 0x7fcec0033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fcec80096a0 tx=0x7fcec8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.188+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcecc01b4a0 con 0x7fced80f5480 2024-02-20T13:26:43.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.395+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fced80faf70 con 0x7fced80f5480 2024-02-20T13:26:43.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.395+0000 7fced4ff9700 1 -- 172.21.15.114:0/2602710864 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcecc01b2c0 con 0x7fced80f5480 2024-02-20T13:26:43.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:43.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcec0031090 msgr2=0x7fcec0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcec0031090 0x7fcec0033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fcec80096a0 tx=0x7fcec8006b40).stop 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 msgr2=0x7fced81056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced81056f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fcecc000c00 tx=0x7fcecc00b040).stop 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 shutdown_connections 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fcec0031090 0x7fcec0033550 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 --2- 172.21.15.114:0/2602710864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fced80f5480 0x7fced81056f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 >> 172.21.15.114:0/2602710864 conn(0x7fced80f0be0 msgr2=0x7fced80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 shutdown_connections 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:43.397+0000 7fcede98f700 1 -- 172.21.15.114:0/2602710864 wait complete. 2024-02-20T13:26:43.398 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:43.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:43 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:42.228242+0000 mgr.smithi038.powepi (mgr.14156) 288 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:44.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:43.396052+0000 mon.smithi038 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.114:0/2602710864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:45.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:45.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:45.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:44.228509+0000 mgr.smithi038.powepi (mgr.14156) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.086+0000 7fa7132d8700 1 -- 172.21.15.114:0/66543690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 msgr2=0x7fa70c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.086+0000 7fa7132d8700 1 --2- 172.21.15.114:0/66543690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c0f58a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa6fc009fb0 tx=0x7fa6fc009b30).stop 2024-02-20T13:26:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.086+0000 7fa7132d8700 1 -- 172.21.15.114:0/66543690 shutdown_connections 2024-02-20T13:26:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.086+0000 7fa7132d8700 1 --2- 172.21.15.114:0/66543690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c0f58a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 -- 172.21.15.114:0/66543690 >> 172.21.15.114:0/66543690 conn(0x7fa70c0f0be0 msgr2=0x7fa70c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 -- 172.21.15.114:0/66543690 shutdown_connections 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 -- 172.21.15.114:0/66543690 wait complete. 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 Processor -- start 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 -- start start 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa7132d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c106cc0 con 0x7fa70c0f5480 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa711074700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa711074700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:39940/0 (socket says 172.21.15.114:39940) 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.087+0000 7fa711074700 1 -- 172.21.15.114:0/1597774865 learned_addr learned my addr 172.21.15.114:0/1597774865 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa711074700 1 -- 172.21.15.114:0/1597774865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc005040 con 0x7fa70c0f5480 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa711074700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c1067d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fa6fc0089f0 tx=0x7fa6fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6fc009320 con 0x7fa70c0f5480 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa6fc013ec0 con 0x7fa70c0f5480 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c106ec0 con 0x7fa70c0f5480 2024-02-20T13:26:47.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6fc014d30 con 0x7fa70c0f5480 2024-02-20T13:26:47.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.088+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c107350 con 0x7fa70c0f5480 2024-02-20T13:26:47.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.089+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa6fc0095f0 con 0x7fa70c0f5480 2024-02-20T13:26:47.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.089+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c045bb0 con 0x7fa70c0f5480 2024-02-20T13:26:47.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.089+0000 7fa701ffb700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa6f8031090 0x7fa6f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:47.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.089+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa6fc0423e0 con 0x7fa70c0f5480 2024-02-20T13:26:47.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.089+0000 7fa710873700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa6f8031090 0x7fa6f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:47.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.090+0000 7fa710873700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa6f8031090 0x7fa6f8033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fa7080096a0 tx=0x7fa708006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:47.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.094+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa6fc01d7f0 con 0x7fa70c0f5480 2024-02-20T13:26:47.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.299+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa70c108200 con 0x7fa70c0f5480 2024-02-20T13:26:47.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.300+0000 7fa701ffb700 1 -- 172.21.15.114:0/1597774865 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6fc01d7f0 con 0x7fa70c0f5480 2024-02-20T13:26:47.301 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:47.301 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:47.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa6f8031090 msgr2=0x7fa6f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa6f8031090 0x7fa6f8033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fa7080096a0 tx=0x7fa708006b40).stop 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 msgr2=0x7fa70c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c1067d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fa6fc0089f0 tx=0x7fa6fc00b040).stop 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 shutdown_connections 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa6f8031090 0x7fa6f8033550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 --2- 172.21.15.114:0/1597774865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa70c0f5480 0x7fa70c1067d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 >> 172.21.15.114:0/1597774865 conn(0x7fa70c0f0be0 msgr2=0x7fa70c05fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.301+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 shutdown_connections 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:47.302+0000 7fa7132d8700 1 -- 172.21.15.114:0/1597774865 wait complete. 2024-02-20T13:26:47.303 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:47.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:47 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:46.228916+0000 mgr.smithi038.powepi (mgr.14156) 290 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:48.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:47.300717+0000 mon.smithi038 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.114:0/1597774865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:48.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:48.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:49.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:48.229394+0000 mgr.smithi038.powepi (mgr.14156) 291 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.969+0000 7ff2fca3c700 1 -- 172.21.15.114:0/3989349324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 msgr2=0x7ff2f80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.969+0000 7ff2fca3c700 1 --2- 172.21.15.114:0/3989349324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f80f56f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff2ec009fb0 tx=0x7ff2ec009b30).stop 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.969+0000 7ff2fca3c700 1 -- 172.21.15.114:0/3989349324 shutdown_connections 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.969+0000 7ff2fca3c700 1 --2- 172.21.15.114:0/3989349324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f80f56f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.969+0000 7ff2fca3c700 1 -- 172.21.15.114:0/3989349324 >> 172.21.15.114:0/3989349324 conn(0x7ff2f80f0a10 msgr2=0x7ff2f80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.970+0000 7ff2fca3c700 1 -- 172.21.15.114:0/3989349324 shutdown_connections 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.970+0000 7ff2fca3c700 1 -- 172.21.15.114:0/3989349324 wait complete. 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.970+0000 7ff2fca3c700 1 Processor -- start 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.970+0000 7ff2fca3c700 1 -- start start 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.970+0000 7ff2fca3c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f8109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.970+0000 7ff2fca3c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f8060300 con 0x7ff2f80f52d0 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2f77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f8109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2f77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f8109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:39956/0 (socket says 172.21.15.114:39956) 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2f77fe700 1 -- 172.21.15.114:0/520322707 learned_addr learned my addr 172.21.15.114:0/520322707 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2f77fe700 1 -- 172.21.15.114:0/520322707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec005040 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2f77fe700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f8109360 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7ff2ec0089f0 tx=0x7ff2ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ec009320 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f8060500 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff2ec013ec0 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ec014d30 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.971+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f8109a10 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.972+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff2ec0095f0 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.972+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f80591e0 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.972+0000 7ff2dffff700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff2e0031090 0x7ff2e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.972+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff2ec0423e0 con 0x7ff2f80f52d0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.972+0000 7ff2f6ffd700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff2e0031090 0x7ff2e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.973+0000 7ff2f6ffd700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff2e0031090 0x7ff2e0033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff2e80096a0 tx=0x7ff2e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:50.977+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff2ec01d7f0 con 0x7ff2f80f52d0 2024-02-20T13:26:51.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.183+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f8045bb0 con 0x7ff2f80f52d0 2024-02-20T13:26:51.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.184+0000 7ff2dffff700 1 -- 172.21.15.114:0/520322707 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2ec01d7f0 con 0x7ff2f80f52d0 2024-02-20T13:26:51.185 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:51.185 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:51.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff2e0031090 msgr2=0x7ff2e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:51.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff2e0031090 0x7ff2e0033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff2e80096a0 tx=0x7ff2e8006b40).stop 2024-02-20T13:26:51.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 msgr2=0x7ff2f8109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:51.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f8109360 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7ff2ec0089f0 tx=0x7ff2ec00b040).stop 2024-02-20T13:26:51.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 shutdown_connections 2024-02-20T13:26:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff2e0031090 0x7ff2e0033550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 --2- 172.21.15.114:0/520322707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2f80f52d0 0x7ff2f8109360 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.185+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 >> 172.21.15.114:0/520322707 conn(0x7ff2f80f0a10 msgr2=0x7ff2f80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.186+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 shutdown_connections 2024-02-20T13:26:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:51.186+0000 7ff2fca3c700 1 -- 172.21.15.114:0/520322707 wait complete. 2024-02-20T13:26:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:51.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:50.229764+0000 mgr.smithi038.powepi (mgr.14156) 292 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:51.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:51 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:51.184674+0000 mon.smithi038 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.114:0/520322707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:52.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:52.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:53.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:52.230112+0000 mgr.smithi038.powepi (mgr.14156) 293 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 -- 172.21.15.114:0/2057910694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 msgr2=0x7fb8640f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 --2- 172.21.15.114:0/2057910694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb8640f9160 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fb854009fb0 tx=0x7fb854009b30).stop 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 -- 172.21.15.114:0/2057910694 shutdown_connections 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 --2- 172.21.15.114:0/2057910694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb8640f9160 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 -- 172.21.15.114:0/2057910694 >> 172.21.15.114:0/2057910694 conn(0x7fb8640f0be0 msgr2=0x7fb8640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 -- 172.21.15.114:0/2057910694 shutdown_connections 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.162+0000 7fb86bcf5700 1 -- 172.21.15.114:0/2057910694 wait complete. 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb86bcf5700 1 Processor -- start 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb86bcf5700 1 -- start start 2024-02-20T13:26:55.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb86bcf5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb864106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb86bcf5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864106c40 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb869a91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb864106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb869a91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb864106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:39986/0 (socket says 172.21.15.114:39986) 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb869a91700 1 -- 172.21.15.114:0/713975251 learned_addr learned my addr 172.21.15.114:0/713975251 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb869a91700 1 -- 172.21.15.114:0/713975251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854005040 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.163+0000 7fb869a91700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb864106750 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fb854009f80 tx=0x7fb85400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.164+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb854009320 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.164+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb864106e40 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.164+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb854013680 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.164+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb85401eda0 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.164+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8641072d0 con 0x7fb864060360 2024-02-20T13:26:55.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.164+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb85401e420 con 0x7fb864060360 2024-02-20T13:26:55.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.165+0000 7fb85a7fc700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb850031090 0x7fb850033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:55.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.165+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb864045bb0 con 0x7fb864060360 2024-02-20T13:26:55.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.165+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb8540423e0 con 0x7fb864060360 2024-02-20T13:26:55.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.165+0000 7fb869290700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb850031090 0x7fb850033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:55.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.165+0000 7fb869290700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb850031090 0x7fb850033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb8600096a0 tx=0x7fb860006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:55.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.169+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb85401d380 con 0x7fb864060360 2024-02-20T13:26:55.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.375+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8641080f0 con 0x7fb864060360 2024-02-20T13:26:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.375+0000 7fb85a7fc700 1 -- 172.21.15.114:0/713975251 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb854004490 con 0x7fb864060360 2024-02-20T13:26:55.376 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:55.377 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb850031090 msgr2=0x7fb850033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb850031090 0x7fb850033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb8600096a0 tx=0x7fb860006b40).stop 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 msgr2=0x7fb864106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb864106750 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fb854009f80 tx=0x7fb85400b040).stop 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 shutdown_connections 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb850031090 0x7fb850033550 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 --2- 172.21.15.114:0/713975251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb864060360 0x7fb864106750 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 >> 172.21.15.114:0/713975251 conn(0x7fb8640f0be0 msgr2=0x7fb8640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 shutdown_connections 2024-02-20T13:26:55.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:55.377+0000 7fb86bcf5700 1 -- 172.21.15.114:0/713975251 wait complete. 2024-02-20T13:26:55.379 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:55.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:55 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:54.230371+0000 mgr.smithi038.powepi (mgr.14156) 294 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:56.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:55.376110+0000 mon.smithi038 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.114:0/713975251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:26:57.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:26:57.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:26:57.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:56.230730+0000 mgr.smithi038.powepi (mgr.14156) 295 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:26:59.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 -- 172.21.15.114:0/485007442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 msgr2=0x7f07c80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 --2- 172.21.15.114:0/485007442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c80f56f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f07c4009fb0 tx=0x7f07c4009b30).stop 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 -- 172.21.15.114:0/485007442 shutdown_connections 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 --2- 172.21.15.114:0/485007442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c80f56f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 -- 172.21.15.114:0/485007442 >> 172.21.15.114:0/485007442 conn(0x7f07c80f0a30 msgr2=0x7f07c80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 -- 172.21.15.114:0/485007442 shutdown_connections 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.128+0000 7f07cecfe700 1 -- 172.21.15.114:0/485007442 wait complete. 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cecfe700 1 Processor -- start 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cecfe700 1 -- start start 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cecfe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c8109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cecfe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8060320 con 0x7f07c80f52d0 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cdcfc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c8109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:59.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cdcfc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c8109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47924/0 (socket says 172.21.15.114:47924) 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cdcfc700 1 -- 172.21.15.114:0/2031019082 learned_addr learned my addr 172.21.15.114:0/2031019082 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cdcfc700 1 -- 172.21.15.114:0/2031019082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4005040 con 0x7f07c80f52d0 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.129+0000 7f07cdcfc700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c8109310 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f07c40089f0 tx=0x7f07c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4009320 con 0x7f07c80f52d0 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c8060520 con 0x7f07c80f52d0 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f07c4013ec0 con 0x7f07c80f52d0 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4014d30 con 0x7f07c80f52d0 2024-02-20T13:26:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c81099c0 con 0x7f07c80f52d0 2024-02-20T13:26:59.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f07c40095f0 con 0x7f07c80f52d0 2024-02-20T13:26:59.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.130+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c8045bb0 con 0x7f07c80f52d0 2024-02-20T13:26:59.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.131+0000 7f07be7fc700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f07b4031090 0x7f07b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:26:59.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.131+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f07c4042420 con 0x7f07c80f52d0 2024-02-20T13:26:59.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.131+0000 7f07cd4fb700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f07b4031090 0x7f07b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:26:59.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.131+0000 7f07cd4fb700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f07b4031090 0x7f07b4033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f07b80096a0 tx=0x7f07b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:26:59.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.135+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f07c401d7f0 con 0x7f07c80f52d0 2024-02-20T13:26:59.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.341+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07c810a1d0 con 0x7f07c80f52d0 2024-02-20T13:26:59.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.342+0000 7f07be7fc700 1 -- 172.21.15.114:0/2031019082 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07c400fe40 con 0x7f07c80f52d0 2024-02-20T13:26:59.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:26:59.344 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f07b4031090 msgr2=0x7f07b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f07b4031090 0x7f07b4033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f07b80096a0 tx=0x7f07b8006b40).stop 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 msgr2=0x7f07c8109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c8109310 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f07c40089f0 tx=0x7f07c400b040).stop 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 shutdown_connections 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f07b4031090 0x7f07b4033550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 --2- 172.21.15.114:0/2031019082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c80f52d0 0x7f07c8109310 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 >> 172.21.15.114:0/2031019082 conn(0x7f07c80f0a30 msgr2=0x7f07c80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 shutdown_connections 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:26:59.344+0000 7f07cecfe700 1 -- 172.21.15.114:0/2031019082 wait complete. 2024-02-20T13:26:59.346 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:26:59.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:26:59 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:26:58.231084+0000 mgr.smithi038.powepi (mgr.14156) 296 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:00.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:26:59.343731+0000 mon.smithi038 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.114:0/2031019082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:01.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:01.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:00.231376+0000 mgr.smithi038.powepi (mgr.14156) 297 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:02.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 -- 172.21.15.114:0/3725989995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 msgr2=0x7fc7940f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 --2- 172.21.15.114:0/3725989995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7940f69b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fc788004760 tx=0x7fc788009ad0).stop 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 -- 172.21.15.114:0/3725989995 shutdown_connections 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 --2- 172.21.15.114:0/3725989995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7940f69b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 -- 172.21.15.114:0/3725989995 >> 172.21.15.114:0/3725989995 conn(0x7fc7940f1cd0 msgr2=0x7fc7940f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 -- 172.21.15.114:0/3725989995 shutdown_connections 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.942+0000 7fc79ae70700 1 -- 172.21.15.114:0/3725989995 wait complete. 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc79ae70700 1 Processor -- start 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc79ae70700 1 -- start start 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc79ae70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7941067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc79ae70700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794106cb0 con 0x7fc7940f6590 2024-02-20T13:27:02.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc798c0c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7941067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc798c0c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7941067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47948/0 (socket says 172.21.15.114:47948) 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.943+0000 7fc798c0c700 1 -- 172.21.15.114:0/860344822 learned_addr learned my addr 172.21.15.114:0/860344822 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc798c0c700 1 -- 172.21.15.114:0/860344822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc788009d70 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc798c0c700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7941067c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc788004170 tx=0x7fc788009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc78800e040 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc794106eb0 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc788008ef0 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc788014c90 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.944+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc794107340 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.945+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc788014420 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.945+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc794045bb0 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.945+0000 7fc7917fa700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7840310e0 0x7fc7840335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.945+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc788042280 con 0x7fc7940f6590 2024-02-20T13:27:02.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.945+0000 7fc793fff700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7840310e0 0x7fc7840335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:02.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.945+0000 7fc793fff700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7840310e0 0x7fc7840335a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc7800096a0 tx=0x7fc780006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:02.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:02.950+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc788012d00 con 0x7fc7940f6590 2024-02-20T13:27:03.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.157+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7941081f0 con 0x7fc7940f6590 2024-02-20T13:27:03.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.157+0000 7fc7917fa700 1 -- 172.21.15.114:0/860344822 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc788012b20 con 0x7fc7940f6590 2024-02-20T13:27:03.160 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:03.160 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7840310e0 msgr2=0x7fc7840335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7840310e0 0x7fc7840335a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc7800096a0 tx=0x7fc780006b40).stop 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 msgr2=0x7fc7941067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7941067c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc788004170 tx=0x7fc788009ad0).stop 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 shutdown_connections 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fc7840310e0 0x7fc7840335a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 --2- 172.21.15.114:0/860344822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7940f6590 0x7fc7941067c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 >> 172.21.15.114:0/860344822 conn(0x7fc7940f1cd0 msgr2=0x7fc7940f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 shutdown_connections 2024-02-20T13:27:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:03.159+0000 7fc79ae70700 1 -- 172.21.15.114:0/860344822 wait complete. 2024-02-20T13:27:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:03.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:02.231704+0000 mgr.smithi038.powepi (mgr.14156) 298 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:03.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:03 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:03.159110+0000 mon.smithi038 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.114:0/860344822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:04.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:04.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:05.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:04.231987+0000 mgr.smithi038.powepi (mgr.14156) 299 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:06.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 -- 172.21.15.114:0/4015118218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 msgr2=0x7fe5800f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 --2- 172.21.15.114:0/4015118218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe5800f56f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fe574009fb0 tx=0x7fe574009b30).stop 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 -- 172.21.15.114:0/4015118218 shutdown_connections 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 --2- 172.21.15.114:0/4015118218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe5800f56f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 -- 172.21.15.114:0/4015118218 >> 172.21.15.114:0/4015118218 conn(0x7fe5800f0a30 msgr2=0x7fe5800f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 -- 172.21.15.114:0/4015118218 shutdown_connections 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.907+0000 7fe585d50700 1 -- 172.21.15.114:0/4015118218 wait complete. 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe585d50700 1 Processor -- start 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe585d50700 1 -- start start 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe585d50700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe580109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe585d50700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580060320 con 0x7fe5800f52d0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe584d4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe580109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe584d4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe580109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:48660/0 (socket says 172.21.15.114:48660) 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.908+0000 7fe584d4e700 1 -- 172.21.15.114:0/2204804719 learned_addr learned my addr 172.21.15.114:0/2204804719 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe584d4e700 1 -- 172.21.15.114:0/2204804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574005040 con 0x7fe5800f52d0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe584d4e700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe580109310 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fe574000c00 tx=0x7fe57400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5740095f0 con 0x7fe5800f52d0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe580060520 con 0x7fe5800f52d0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5801099c0 con 0x7fe5800f52d0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe574007e50 con 0x7fe5800f52d0 2024-02-20T13:27:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.909+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe574014e00 con 0x7fe5800f52d0 2024-02-20T13:27:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.910+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe574025070 con 0x7fe5800f52d0 2024-02-20T13:27:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.910+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe580045bb0 con 0x7fe5800f52d0 2024-02-20T13:27:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.910+0000 7fe57d7fa700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe570031090 0x7fe570033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.910+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe574041e10 con 0x7fe5800f52d0 2024-02-20T13:27:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.910+0000 7fe57ffff700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe570031090 0x7fe570033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.911+0000 7fe57ffff700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe570031090 0x7fe570033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe56c0096a0 tx=0x7fe56c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:06.915+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe57400f8d0 con 0x7fe5800f52d0 2024-02-20T13:27:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.127+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe58010a1b0 con 0x7fe5800f52d0 2024-02-20T13:27:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.127+0000 7fe57d7fa700 1 -- 172.21.15.114:0/2204804719 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe574043020 con 0x7fe5800f52d0 2024-02-20T13:27:07.130 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:07.130 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.128+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe570031090 msgr2=0x7fe570033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.128+0000 7fe585d50700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe570031090 0x7fe570033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe56c0096a0 tx=0x7fe56c006b40).stop 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 msgr2=0x7fe580109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe580109310 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fe574000c00 tx=0x7fe57400b040).stop 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 shutdown_connections 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fe570031090 0x7fe570033550 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:07.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 --2- 172.21.15.114:0/2204804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5800f52d0 0x7fe580109310 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 >> 172.21.15.114:0/2204804719 conn(0x7fe5800f0a30 msgr2=0x7fe5800f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 shutdown_connections 2024-02-20T13:27:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:07.129+0000 7fe585d50700 1 -- 172.21.15.114:0/2204804719 wait complete. 2024-02-20T13:27:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:07.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:06.232257+0000 mgr.smithi038.powepi (mgr.14156) 300 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:07.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:07 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:07.128886+0000 mon.smithi038 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.114:0/2204804719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:08.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:08.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:09.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:08.232593+0000 mgr.smithi038.powepi (mgr.14156) 301 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 -- 172.21.15.114:0/511661541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 msgr2=0x7f5e34069f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 --2- 172.21.15.114:0/511661541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34069f40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f5e28009fb0 tx=0x7f5e28009b30).stop 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 -- 172.21.15.114:0/511661541 shutdown_connections 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 --2- 172.21.15.114:0/511661541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34069f40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 -- 172.21.15.114:0/511661541 >> 172.21.15.114:0/511661541 conn(0x7f5e340f1b40 msgr2=0x7f5e340f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 -- 172.21.15.114:0/511661541 shutdown_connections 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 -- 172.21.15.114:0/511661541 wait complete. 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.824+0000 7f5e33fff700 1 Processor -- start 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e33fff700 1 -- start start 2024-02-20T13:27:10.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e33fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34108810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e33fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34108d00 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e32ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34108810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e32ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34108810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:48668/0 (socket says 172.21.15.114:48668) 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e32ffd700 1 -- 172.21.15.114:0/18211255 learned_addr learned my addr 172.21.15.114:0/18211255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e32ffd700 1 -- 172.21.15.114:0/18211255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e28005040 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e32ffd700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34108810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f5e28009f80 tx=0x7f5e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e28009320 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.825+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e34108f00 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5e28013680 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e2801eda0 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e34109390 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5e2801e420 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e34045bb0 con 0x7f5e3406ba40 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e1b7fe700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e1c031090 0x7f5e1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:10.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.826+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5e28042470 con 0x7f5e3406ba40 2024-02-20T13:27:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.827+0000 7f5e327fc700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e1c031090 0x7f5e1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.827+0000 7f5e327fc700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e1c031090 0x7f5e1c033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5e240096a0 tx=0x7f5e24006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:10.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:10.831+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5e2801d380 con 0x7f5e3406ba40 2024-02-20T13:27:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.036+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e3410a240 con 0x7f5e3406ba40 2024-02-20T13:27:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.037+0000 7f5e1b7fe700 1 -- 172.21.15.114:0/18211255 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e28004490 con 0x7f5e3406ba40 2024-02-20T13:27:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:11.039 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e1c031090 msgr2=0x7f5e1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e1c031090 0x7f5e1c033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5e240096a0 tx=0x7f5e24006b40).stop 2024-02-20T13:27:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 msgr2=0x7f5e34108810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34108810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f5e28009f80 tx=0x7f5e2800b040).stop 2024-02-20T13:27:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 shutdown_connections 2024-02-20T13:27:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e1c031090 0x7f5e1c033550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 --2- 172.21.15.114:0/18211255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e3406ba40 0x7f5e34108810 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.038+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 >> 172.21.15.114:0/18211255 conn(0x7f5e340f1b40 msgr2=0x7f5e340f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.039+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 shutdown_connections 2024-02-20T13:27:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:11.039+0000 7f5e33fff700 1 -- 172.21.15.114:0/18211255 wait complete. 2024-02-20T13:27:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:11.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:10.232844+0000 mgr.smithi038.powepi (mgr.14156) 302 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:11.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:11 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:11.038504+0000 mon.smithi038 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.114:0/18211255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:12.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:12.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:13.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:12.233074+0000 mgr.smithi038.powepi (mgr.14156) 303 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 -- 172.21.15.114:0/1577227245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 msgr2=0x7fca440f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 --2- 172.21.15.114:0/1577227245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca440f69b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fca34004760 tx=0x7fca34009ad0).stop 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 -- 172.21.15.114:0/1577227245 shutdown_connections 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 --2- 172.21.15.114:0/1577227245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca440f69b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 -- 172.21.15.114:0/1577227245 >> 172.21.15.114:0/1577227245 conn(0x7fca440f1cd0 msgr2=0x7fca440f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 -- 172.21.15.114:0/1577227245 shutdown_connections 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.867+0000 7fca4b3c2700 1 -- 172.21.15.114:0/1577227245 wait complete. 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4b3c2700 1 Processor -- start 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4b3c2700 1 -- start start 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4b3c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca441067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:14.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4b3c2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca44106cb0 con 0x7fca440f6590 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4915e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca441067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4915e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca441067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:48692/0 (socket says 172.21.15.114:48692) 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.868+0000 7fca4915e700 1 -- 172.21.15.114:0/2264596891 learned_addr learned my addr 172.21.15.114:0/2264596891 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca4915e700 1 -- 172.21.15.114:0/2264596891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34009d70 con 0x7fca440f6590 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca4915e700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca441067c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fca34004170 tx=0x7fca34009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3400e040 con 0x7fca440f6590 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca44106eb0 con 0x7fca440f6590 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fca34008ef0 con 0x7fca440f6590 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca34014c90 con 0x7fca440f6590 2024-02-20T13:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.869+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca44107340 con 0x7fca440f6590 2024-02-20T13:27:14.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.870+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fca34014420 con 0x7fca440f6590 2024-02-20T13:27:14.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.870+0000 7fca39ffb700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fca30031090 0x7fca30033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:14.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.870+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fca340422b0 con 0x7fca440f6590 2024-02-20T13:27:14.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.870+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca44045bb0 con 0x7fca440f6590 2024-02-20T13:27:14.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.870+0000 7fca4895d700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fca30031090 0x7fca30033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:14.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.871+0000 7fca4895d700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fca30031090 0x7fca30033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fca400096a0 tx=0x7fca40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:14.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:14.875+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fca34026300 con 0x7fca440f6590 2024-02-20T13:27:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.086+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca440fc080 con 0x7fca440f6590 2024-02-20T13:27:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.087+0000 7fca39ffb700 1 -- 172.21.15.114:0/2264596891 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca34012b20 con 0x7fca440f6590 2024-02-20T13:27:15.089 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:15.089 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fca30031090 msgr2=0x7fca30033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fca30031090 0x7fca30033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fca400096a0 tx=0x7fca40006b40).stop 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 msgr2=0x7fca441067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca441067c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fca34004170 tx=0x7fca34009ad0).stop 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 shutdown_connections 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fca30031090 0x7fca30033550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 --2- 172.21.15.114:0/2264596891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca440f6590 0x7fca441067c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 >> 172.21.15.114:0/2264596891 conn(0x7fca440f1cd0 msgr2=0x7fca440f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 shutdown_connections 2024-02-20T13:27:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:15.088+0000 7fca4b3c2700 1 -- 172.21.15.114:0/2264596891 wait complete. 2024-02-20T13:27:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:15.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:14.233483+0000 mgr.smithi038.powepi (mgr.14156) 304 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:15.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:15 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:15.088219+0000 mon.smithi038 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.114:0/2264596891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:16.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:16.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:17.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:16.233870+0000 mgr.smithi038.powepi (mgr.14156) 305 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:18.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 -- 172.21.15.114:0/1133343946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 msgr2=0x7f85400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:18.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 --2- 172.21.15.114:0/1133343946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85400f58a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f8530009fb0 tx=0x7f8530009b30).stop 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 -- 172.21.15.114:0/1133343946 shutdown_connections 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 --2- 172.21.15.114:0/1133343946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85400f58a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 -- 172.21.15.114:0/1133343946 >> 172.21.15.114:0/1133343946 conn(0x7f85400f0be0 msgr2=0x7f85400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 -- 172.21.15.114:0/1133343946 shutdown_connections 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.604+0000 7f854838c700 1 -- 172.21.15.114:0/1133343946 wait complete. 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f854838c700 1 Processor -- start 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f854838c700 1 -- start start 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f854838c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85401094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f854838c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8540060320 con 0x7f85400f5480 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f8546128700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85401094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f8546128700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85401094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54342/0 (socket says 172.21.15.114:54342) 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.605+0000 7f8546128700 1 -- 172.21.15.114:0/4121165263 learned_addr learned my addr 172.21.15.114:0/4121165263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f8546128700 1 -- 172.21.15.114:0/4121165263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530005040 con 0x7f85400f5480 2024-02-20T13:27:18.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f8546128700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85401094f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f85300089f0 tx=0x7f853000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8530009320 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8540060520 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8530013ec0 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8530014d30 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.606+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8540109ba0 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.607+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f85300095f0 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.607+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8540045bb0 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.607+0000 7f8536ffd700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f852c031090 0x7f852c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.607+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f85300423e0 con 0x7f85400f5480 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.607+0000 7f8545927700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f852c031090 0x7f852c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.607+0000 7f8545927700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f852c031090 0x7f852c033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f853c0096a0 tx=0x7f853c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:18.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.612+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f853001d7f0 con 0x7f85400f5480 2024-02-20T13:27:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.821+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85400faf70 con 0x7f85400f5480 2024-02-20T13:27:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.821+0000 7f8536ffd700 1 -- 172.21.15.114:0/4121165263 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f853001d7f0 con 0x7f85400f5480 2024-02-20T13:27:18.822 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:18.822 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f852c031090 msgr2=0x7f852c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f852c031090 0x7f852c033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f853c0096a0 tx=0x7f853c006b40).stop 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 msgr2=0x7f85401094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85401094f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f85300089f0 tx=0x7f853000b040).stop 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 shutdown_connections 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f852c031090 0x7f852c033550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 --2- 172.21.15.114:0/4121165263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85400f5480 0x7f85401094f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 >> 172.21.15.114:0/4121165263 conn(0x7f85400f0be0 msgr2=0x7f85400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 shutdown_connections 2024-02-20T13:27:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:18.823+0000 7f854838c700 1 -- 172.21.15.114:0/4121165263 wait complete. 2024-02-20T13:27:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:19.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:18.234219+0000 mgr.smithi038.powepi (mgr.14156) 306 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:19.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:19 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:18.822035+0000 mon.smithi038 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.114:0/4121165263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:20.508 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:20.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:21.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:20.234652+0000 mgr.smithi038.powepi (mgr.14156) 307 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:22.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 -- 172.21.15.114:0/3750797189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 msgr2=0x7f5ea00f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 --2- 172.21.15.114:0/3750797189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea00f4c60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5e90009960 tx=0x7f5e90004d10).stop 2024-02-20T13:27:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 -- 172.21.15.114:0/3750797189 shutdown_connections 2024-02-20T13:27:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 --2- 172.21.15.114:0/3750797189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea00f4c60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 -- 172.21.15.114:0/3750797189 >> 172.21.15.114:0/3750797189 conn(0x7f5ea00f0360 msgr2=0x7f5ea00f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 -- 172.21.15.114:0/3750797189 shutdown_connections 2024-02-20T13:27:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.647+0000 7f5ea5af2700 1 -- 172.21.15.114:0/3750797189 wait complete. 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5ea5af2700 1 Processor -- start 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5ea5af2700 1 -- start start 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5ea5af2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5ea5af2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea0106cb0 con 0x7f5ea00f4840 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5e9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5e9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:54366/0 (socket says 172.21.15.114:54366) 2024-02-20T13:27:22.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5e9f7fe700 1 -- 172.21.15.114:0/843713648 learned_addr learned my addr 172.21.15.114:0/843713648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.648+0000 7f5e9f7fe700 1 -- 172.21.15.114:0/843713648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e90009640 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.649+0000 7f5e9f7fe700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea01067c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f5e90009610 tx=0x7f5e90004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.649+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e90018070 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.649+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea0106eb0 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.649+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5e9000b7e0 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.649+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9001d420 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.649+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea0107340 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.650+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5e90024060 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.650+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea0045bb0 con 0x7f5ea00f4840 2024-02-20T13:27:22.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.650+0000 7f5e87fff700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e88031090 0x7f5e88033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:22.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.650+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5e90042670 con 0x7f5ea00f4840 2024-02-20T13:27:22.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.650+0000 7f5e9effd700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e88031090 0x7f5e88033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:22.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.651+0000 7f5e9effd700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e88031090 0x7f5e88033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:22.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.655+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5e900116a0 con 0x7f5ea00f4840 2024-02-20T13:27:22.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.865+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ea0108190 con 0x7f5ea00f4840 2024-02-20T13:27:22.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.866+0000 7f5e87fff700 1 -- 172.21.15.114:0/843713648 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e900118b0 con 0x7f5ea00f4840 2024-02-20T13:27:22.867 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:22.867 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:22.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.867+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e88031090 msgr2=0x7f5e88033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:22.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e88031090 0x7f5e88033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).stop 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 msgr2=0x7f5ea01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea01067c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f5e90009610 tx=0x7f5e90004d10).stop 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 shutdown_connections 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f5e88031090 0x7f5e88033550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 --2- 172.21.15.114:0/843713648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00f4840 0x7f5ea01067c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 >> 172.21.15.114:0/843713648 conn(0x7f5ea00f0360 msgr2=0x7f5ea00f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 shutdown_connections 2024-02-20T13:27:22.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:22.868+0000 7f5ea5af2700 1 -- 172.21.15.114:0/843713648 wait complete. 2024-02-20T13:27:22.870 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:22.234951+0000 mgr.smithi038.powepi (mgr.14156) 308 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:23.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:23 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:22.866853+0000 mon.smithi038 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.114:0/843713648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:24.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:24.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:25.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:24.235316+0000 mgr.smithi038.powepi (mgr.14156) 309 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:26.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 -- 172.21.15.114:0/624135105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 msgr2=0x7f56d00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 --2- 172.21.15.114:0/624135105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d00f58a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f56cc009fb0 tx=0x7f56cc009b30).stop 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 -- 172.21.15.114:0/624135105 shutdown_connections 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 --2- 172.21.15.114:0/624135105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d00f58a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 -- 172.21.15.114:0/624135105 >> 172.21.15.114:0/624135105 conn(0x7f56d00f0be0 msgr2=0x7f56d00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 -- 172.21.15.114:0/624135105 shutdown_connections 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 -- 172.21.15.114:0/624135105 wait complete. 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 Processor -- start 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.467+0000 7f56d8464700 1 -- start start 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d8464700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d8464700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0105be0 con 0x7f56d00f5480 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d6200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d6200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:36670/0 (socket says 172.21.15.114:36670) 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d6200700 1 -- 172.21.15.114:0/1134475633 learned_addr learned my addr 172.21.15.114:0/1134475633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:26.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d6200700 1 -- 172.21.15.114:0/1134475633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc005040 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.468+0000 7f56d6200700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d01056f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f56cc0089f0 tx=0x7f56cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56cc009320 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d0105de0 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f56cc013ec0 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56cc014d30 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d0106270 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f56cc014420 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.469+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d0045bb0 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.470+0000 7f56c6ffd700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f56bc031090 0x7f56bc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.470+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f56cc0424c0 con 0x7f56d00f5480 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.470+0000 7f56d59ff700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f56bc031090 0x7f56bc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:26.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.470+0000 7f56d59ff700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f56bc031090 0x7f56bc033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f56c00096a0 tx=0x7f56c0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:26.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.474+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f56cc01d380 con 0x7f56d00f5480 2024-02-20T13:27:26.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.682+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56d00faf70 con 0x7f56d00f5480 2024-02-20T13:27:26.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.682+0000 7f56c6ffd700 1 -- 172.21.15.114:0/1134475633 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56cc01d380 con 0x7f56d00f5480 2024-02-20T13:27:26.683 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:26.683 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f56bc031090 msgr2=0x7f56bc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f56bc031090 0x7f56bc033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f56c00096a0 tx=0x7f56c0006b40).stop 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 msgr2=0x7f56d01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d01056f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f56cc0089f0 tx=0x7f56cc00b040).stop 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 shutdown_connections 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f56bc031090 0x7f56bc033550 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 --2- 172.21.15.114:0/1134475633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d00f5480 0x7f56d01056f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 >> 172.21.15.114:0/1134475633 conn(0x7f56d00f0be0 msgr2=0x7f56d00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 shutdown_connections 2024-02-20T13:27:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:26.684+0000 7f56d8464700 1 -- 172.21.15.114:0/1134475633 wait complete. 2024-02-20T13:27:26.686 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:26.235621+0000 mgr.smithi038.powepi (mgr.14156) 310 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:27.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:27 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:26.683142+0000 mon.smithi038 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.114:0/1134475633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:28.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:28.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:28.285255+0000 mon.smithi038 (mon.0) 285 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:27:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:28.285487+0000 mon.smithi038 (mon.0) 286 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:27:29.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:28.235886+0000 mgr.smithi038.powepi (mgr.14156) 311 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/3428793220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 msgr2=0x7f1b140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 --2- 172.21.15.114:0/3428793220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b140f58a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f1b04009fb0 tx=0x7f1b04009b30).stop 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/3428793220 shutdown_connections 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 --2- 172.21.15.114:0/3428793220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b140f58a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/3428793220 >> 172.21.15.114:0/3428793220 conn(0x7f1b140f0be0 msgr2=0x7f1b140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/3428793220 shutdown_connections 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.572+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/3428793220 wait complete. 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b1b3d7700 1 Processor -- start 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b1b3d7700 1 -- start start 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b1b3d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b141056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b1b3d7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14105be0 con 0x7f1b140f5480 2024-02-20T13:27:30.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b19173700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b141056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b19173700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b141056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:36684/0 (socket says 172.21.15.114:36684) 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.573+0000 7f1b19173700 1 -- 172.21.15.114:0/2468111117 learned_addr learned my addr 172.21.15.114:0/2468111117 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b19173700 1 -- 172.21.15.114:0/2468111117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04005040 con 0x7f1b140f5480 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b19173700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b141056f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f1b040089f0 tx=0x7f1b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b04009320 con 0x7f1b140f5480 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b14105de0 con 0x7f1b140f5480 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1b04013ec0 con 0x7f1b140f5480 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b04014d30 con 0x7f1b140f5480 2024-02-20T13:27:30.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.574+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b14106270 con 0x7f1b140f5480 2024-02-20T13:27:30.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.575+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1b040095f0 con 0x7f1b140f5480 2024-02-20T13:27:30.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.575+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b14045bb0 con 0x7f1b140f5480 2024-02-20T13:27:30.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.575+0000 7f1b09ffb700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1b000310e0 0x7f1b000335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:30.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.575+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1b04044260 con 0x7f1b140f5480 2024-02-20T13:27:30.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.575+0000 7f1b18972700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1b000310e0 0x7f1b000335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:30.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.576+0000 7f1b18972700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1b000310e0 0x7f1b000335a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f1b100096a0 tx=0x7f1b10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:30.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.580+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1b0400f930 con 0x7f1b140f5480 2024-02-20T13:27:30.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.788+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b140faf70 con 0x7f1b140f5480 2024-02-20T13:27:30.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.789+0000 7f1b09ffb700 1 -- 172.21.15.114:0/2468111117 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b0401f7f0 con 0x7f1b140f5480 2024-02-20T13:27:30.790 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:30.790 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:30.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.790+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1b000310e0 msgr2=0x7f1b000335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:30.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.790+0000 7f1b1b3d7700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1b000310e0 0x7f1b000335a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f1b100096a0 tx=0x7f1b10006b40).stop 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.790+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 msgr2=0x7f1b141056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.790+0000 7f1b1b3d7700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b141056f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f1b040089f0 tx=0x7f1b0400b040).stop 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.791+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 shutdown_connections 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.791+0000 7f1b1b3d7700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1b000310e0 0x7f1b000335a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.791+0000 7f1b1b3d7700 1 --2- 172.21.15.114:0/2468111117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b140f5480 0x7f1b141056f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.791+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 >> 172.21.15.114:0/2468111117 conn(0x7f1b140f0be0 msgr2=0x7f1b140f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.791+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 shutdown_connections 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:30.791+0000 7f1b1b3d7700 1 -- 172.21.15.114:0/2468111117 wait complete. 2024-02-20T13:27:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:31.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:30.236174+0000 mgr.smithi038.powepi (mgr.14156) 312 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:31.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:31 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:30.789640+0000 mon.smithi038 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.114:0/2468111117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:32.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:32.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:32.583 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:31.480114+0000 mon.smithi038 (mon.0) 288 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T13:27:32.584 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:31.481360+0000 mon.smithi038 (mon.0) 289 : audit [DBG] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T13:27:32.584 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:31.482207+0000 mon.smithi038 (mon.0) 290 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T13:27:32.584 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:31.722529+0000 mon.smithi038 (mon.0) 291 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:32.584 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:31.922433+0000 mon.smithi038 (mon.0) 292 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:32.584 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:32.158299+0000 mon.smithi038 (mon.0) 293 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:27:31.479066+0000 mgr.smithi038.powepi (mgr.14156) 313 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-prometheus.smithi038 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-prometheus.smithi038 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon prometheus.smithi038 ... 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:25:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:26:16Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:26:54Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:27:33.697 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4269, in extract_uid_gid_monitoring 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('uid/gid not found') 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: uid/gid not found 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:27:33.698 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: yield (conn, connr) 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: code, '\n'.join(err))) 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-prometheus.smithi038 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-prometheus.smithi038 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon prometheus.smithi038 ... 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:25:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:26:16Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:26:54Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:27:33.699 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:27:33.700 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4269, in extract_uid_gid_monitoring 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('uid/gid not found') 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: uid/gid not found 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:27:31.479299+0000 mgr.smithi038.powepi (mgr.14156) 314 : cephadm [ERR] Failed while placing prometheus.smithi038 on smithi038: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-24090638-cff2-11ee-95bb-87774f69a715-prometheus.smithi038 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: /bin/podman: stderr Error: no such container ceph-24090638-cff2-11ee-95bb-87774f69a715-prometheus.smithi038 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Deploy daemon prometheus.smithi038 ... 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:25:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:26:16Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr time="2024-02-20T13:26:54Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T13:27:33.701 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: Traceback (most recent call last): 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: main() 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: r = ctx.func(ctx) 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: return func(ctx) 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4269, in extract_uid_gid_monitoring 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: File "/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: raise RuntimeError('uid/gid not found') 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: RuntimeError: uid/gid not found 2024-02-20T13:27:33.702 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:32.236435+0000 mgr.smithi038.powepi (mgr.14156) 315 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:33.703 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:33.302029+0000 mon.smithi038 (mon.0) 294 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:35.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:33.917084+0000 mon.smithi038 (mon.0) 295 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:35.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:27:34.170460+0000 mgr.smithi038.powepi (mgr.14156) 316 : cephadm [INF] Updating smithi038:/etc/ceph/ceph.conf 2024-02-20T13:27:35.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:27:34.193056+0000 mgr.smithi038.powepi (mgr.14156) 317 : cephadm [INF] Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-02-20T13:27:35.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:34.225700+0000 mon.smithi038 (mon.0) 296 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:35.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:34.236714+0000 mgr.smithi038.powepi (mgr.14156) 318 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:36.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.370+0000 7f28d4315700 1 -- 172.21.15.114:0/2456748352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 msgr2=0x7f28cc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.370+0000 7f28d4315700 1 --2- 172.21.15.114:0/2456748352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc0f69b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f28bc004760 tx=0x7f28bc009ad0).stop 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 -- 172.21.15.114:0/2456748352 shutdown_connections 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 --2- 172.21.15.114:0/2456748352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc0f69b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 -- 172.21.15.114:0/2456748352 >> 172.21.15.114:0/2456748352 conn(0x7f28cc0f1cd0 msgr2=0x7f28cc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 -- 172.21.15.114:0/2456748352 shutdown_connections 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 -- 172.21.15.114:0/2456748352 wait complete. 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 Processor -- start 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 -- start start 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d4315700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc106cb0 con 0x7f28cc0f6590 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d20b1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:36.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d20b1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44268/0 (socket says 172.21.15.114:44268) 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.371+0000 7f28d20b1700 1 -- 172.21.15.114:0/3813586615 learned_addr learned my addr 172.21.15.114:0/3813586615 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28d20b1700 1 -- 172.21.15.114:0/3813586615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc009d70 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28d20b1700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc1067c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f28bc004170 tx=0x7f28bc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28bc00e040 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc106eb0 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f28bc008ef0 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28cc107340 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.372+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28bc014c90 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.373+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f28bc0283f0 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.373+0000 7f28c2ffd700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f28b8031130 0x7f28b80335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.373+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f28bc042290 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.373+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28cc045bb0 con 0x7f28cc0f6590 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.373+0000 7f28d18b0700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f28b8031130 0x7f28b80335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:36.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.374+0000 7f28d18b0700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f28b8031130 0x7f28b80335f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f28c80096a0 tx=0x7f28c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:36.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.378+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f28bc0268f0 con 0x7f28cc0f6590 2024-02-20T13:27:36.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.586+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28cc0fc170 con 0x7f28cc0f6590 2024-02-20T13:27:36.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.587+0000 7f28c2ffd700 1 -- 172.21.15.114:0/3813586615 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f28bc026300 con 0x7f28cc0f6590 2024-02-20T13:27:36.587 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:36.587 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f28b8031130 msgr2=0x7f28b80335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f28b8031130 0x7f28b80335f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f28c80096a0 tx=0x7f28c8006b40).stop 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 msgr2=0x7f28cc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc1067c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f28bc004170 tx=0x7f28bc009ad0).stop 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 shutdown_connections 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f28b8031130 0x7f28b80335f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 --2- 172.21.15.114:0/3813586615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28cc0f6590 0x7f28cc1067c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 >> 172.21.15.114:0/3813586615 conn(0x7f28cc0f1cd0 msgr2=0x7f28cc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:36.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 shutdown_connections 2024-02-20T13:27:36.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:36.588+0000 7f28d4315700 1 -- 172.21.15.114:0/3813586615 wait complete. 2024-02-20T13:27:36.590 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:37.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:36.166322+0000 mon.smithi038 (mon.0) 297 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:37.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:36.237036+0000 mgr.smithi038.powepi (mgr.14156) 319 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:37.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:36.587395+0000 mon.smithi038 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.114:0/3813586615' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:37.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:37.002779+0000 mon.smithi038 (mon.0) 299 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:37.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:37.003515+0000 mon.smithi038 (mon.0) 300 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-02-20T13:27:37.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:37.007177+0000 mon.smithi038 (mon.0) 301 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' 2024-02-20T13:27:38.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:38.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:38.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:38 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:37.004732+0000 mgr.smithi038.powepi (mgr.14156) 320 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:38.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:38 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cephadm 2024-02-20T13:27:37.010128+0000 mgr.smithi038.powepi (mgr.14156) 321 : cephadm [INF] Deploying daemon alertmanager.smithi038 on smithi038 2024-02-20T13:27:38.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:38 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:37.164869+0000 mon.smithi038 (mon.0) 302 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/2434596082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 msgr2=0x7f7d480f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 --2- 172.21.15.114:0/2434596082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d480f58d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7d38009fb0 tx=0x7f7d38009b30).stop 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/2434596082 shutdown_connections 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 --2- 172.21.15.114:0/2434596082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d480f58d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/2434596082 >> 172.21.15.114:0/2434596082 conn(0x7f7d480f0c10 msgr2=0x7f7d480f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/2434596082 shutdown_connections 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/2434596082 wait complete. 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 Processor -- start 2024-02-20T13:27:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- start start 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d481067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.392+0000 7f7d4e7ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d48106ca0 con 0x7f7d480f54b0 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d47fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d481067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d47fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d481067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44276/0 (socket says 172.21.15.114:44276) 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d47fff700 1 -- 172.21.15.114:0/3575857227 learned_addr learned my addr 172.21.15.114:0/3575857227 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d47fff700 1 -- 172.21.15.114:0/3575857227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38005040 con 0x7f7d480f54b0 2024-02-20T13:27:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d47fff700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d481067b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f7d380089f0 tx=0x7f7d3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d38009320 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d48106ea0 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7d38013ec0 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d38014d30 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.393+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d48107330 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.394+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7d38014420 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.394+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d48045bb0 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.394+0000 7f7d44ff9700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7d30031090 0x7f7d30033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.394+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7d380424a0 con 0x7f7d480f54b0 2024-02-20T13:27:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.395+0000 7f7d477fe700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7d30031090 0x7f7d30033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:40.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.395+0000 7f7d477fe700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7d30031090 0x7f7d30033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f7d3c0096a0 tx=0x7f7d3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:40.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.399+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7d3801d7f0 con 0x7f7d480f54b0 2024-02-20T13:27:40.446 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:39.005047+0000 mgr.smithi038.powepi (mgr.14156) 322 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:40.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.605+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d48108180 con 0x7f7d480f54b0 2024-02-20T13:27:40.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.605+0000 7f7d44ff9700 1 -- 172.21.15.114:0/3575857227 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d3801d7f0 con 0x7f7d480f54b0 2024-02-20T13:27:40.607 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:40.608 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:40.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7d30031090 msgr2=0x7f7d30033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:40.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7d30031090 0x7f7d30033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f7d3c0096a0 tx=0x7f7d3c006b40).stop 2024-02-20T13:27:40.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 msgr2=0x7f7d481067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:40.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d481067b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f7d380089f0 tx=0x7f7d3800b040).stop 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 shutdown_connections 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f7d30031090 0x7f7d30033550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 --2- 172.21.15.114:0/3575857227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d480f54b0 0x7f7d481067b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 >> 172.21.15.114:0/3575857227 conn(0x7f7d480f0c10 msgr2=0x7f7d480f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 shutdown_connections 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:40.607+0000 7f7d4e7ae700 1 -- 172.21.15.114:0/3575857227 wait complete. 2024-02-20T13:27:40.610 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:41.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:40.606052+0000 mon.smithi038 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.114:0/3575857227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:42.434 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:42.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:42.445 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:42 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:41.005476+0000 mgr.smithi038.powepi (mgr.14156) 323 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- 172.21.15.114:0/3600884024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 msgr2=0x7f136c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 --2- 172.21.15.114:0/3600884024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c0f69b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f135c004760 tx=0x7f135c009ad0).stop 2024-02-20T13:27:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- 172.21.15.114:0/3600884024 shutdown_connections 2024-02-20T13:27:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 --2- 172.21.15.114:0/3600884024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c0f69b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- 172.21.15.114:0/3600884024 >> 172.21.15.114:0/3600884024 conn(0x7f136c0f1cd0 msgr2=0x7f136c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- 172.21.15.114:0/3600884024 shutdown_connections 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- 172.21.15.114:0/3600884024 wait complete. 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 Processor -- start 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- start start 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.391+0000 7f137448a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c106cb0 con 0x7f136c0f6590 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1372226700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1372226700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44284/0 (socket says 172.21.15.114:44284) 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1372226700 1 -- 172.21.15.114:0/2101795057 learned_addr learned my addr 172.21.15.114:0/2101795057 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1372226700 1 -- 172.21.15.114:0/2101795057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c009d70 con 0x7f136c0f6590 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1372226700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c1067c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f135c004170 tx=0x7f135c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f135c00e040 con 0x7f136c0f6590 2024-02-20T13:27:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c106eb0 con 0x7f136c0f6590 2024-02-20T13:27:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c107340 con 0x7f136c0f6590 2024-02-20T13:27:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.392+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f135c008ef0 con 0x7f136c0f6590 2024-02-20T13:27:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.393+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f135c014c90 con 0x7f136c0f6590 2024-02-20T13:27:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.393+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f135c014420 con 0x7f136c0f6590 2024-02-20T13:27:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.393+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c045bb0 con 0x7f136c0f6590 2024-02-20T13:27:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.393+0000 7f1362ffd700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1358031090 0x7f1358033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.393+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f135c042260 con 0x7f136c0f6590 2024-02-20T13:27:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.394+0000 7f1371a25700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1358031090 0x7f1358033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.394+0000 7f1371a25700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1358031090 0x7f1358033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f13680096a0 tx=0x7f1368006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:44.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.398+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f135c026300 con 0x7f136c0f6590 2024-02-20T13:27:44.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.606+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f136c0fc080 con 0x7f136c0f6590 2024-02-20T13:27:44.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.606+0000 7f1362ffd700 1 -- 172.21.15.114:0/2101795057 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f135c012b20 con 0x7f136c0f6590 2024-02-20T13:27:44.608 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:44.608 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1358031090 msgr2=0x7f1358033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1358031090 0x7f1358033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f13680096a0 tx=0x7f1368006b40).stop 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 msgr2=0x7f136c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c1067c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f135c004170 tx=0x7f135c009ad0).stop 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 shutdown_connections 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1358031090 0x7f1358033550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 --2- 172.21.15.114:0/2101795057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f136c0f6590 0x7f136c1067c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 >> 172.21.15.114:0/2101795057 conn(0x7f136c0f1cd0 msgr2=0x7f136c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 shutdown_connections 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:44.608+0000 7f137448a700 1 -- 172.21.15.114:0/2101795057 wait complete. 2024-02-20T13:27:44.610 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:44.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:43.005834+0000 mgr.smithi038.powepi (mgr.14156) 324 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:45.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:44.608142+0000 mon.smithi038 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.114:0/2101795057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:46.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:46.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:46.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:45.006143+0000 mgr.smithi038.powepi (mgr.14156) 325 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:48.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 -- 172.21.15.114:0/3656751096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 msgr2=0x7f700c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:48.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 --2- 172.21.15.114:0/3656751096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c0f56f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f6ffc009fb0 tx=0x7f6ffc009b30).stop 2024-02-20T13:27:48.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 -- 172.21.15.114:0/3656751096 shutdown_connections 2024-02-20T13:27:48.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 --2- 172.21.15.114:0/3656751096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c0f56f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 -- 172.21.15.114:0/3656751096 >> 172.21.15.114:0/3656751096 conn(0x7f700c0f0a30 msgr2=0x7f700c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 -- 172.21.15.114:0/3656751096 shutdown_connections 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 -- 172.21.15.114:0/3656751096 wait complete. 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.276+0000 7f700bfff700 1 Processor -- start 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700bfff700 1 -- start start 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700bfff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c105a00 con 0x7f700c0f52d0 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:40448/0 (socket says 172.21.15.114:40448) 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700affd700 1 -- 172.21.15.114:0/2175093882 learned_addr learned my addr 172.21.15.114:0/2175093882 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700affd700 1 -- 172.21.15.114:0/2175093882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc005040 con 0x7f700c0f52d0 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.277+0000 7f700affd700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c105510 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6ffc0089f0 tx=0x7f6ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.278+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ffc009320 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.278+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c105c00 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.278+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6ffc013ec0 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.278+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ffc014d30 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.278+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700c106090 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.278+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6ffc014420 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.279+0000 7f6ff37fe700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6ff4031090 0x7f6ff4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.279+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700c045bb0 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.279+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6ffc0423f0 con 0x7f700c0f52d0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.279+0000 7f700a7fc700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6ff4031090 0x7f6ff4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.279+0000 7f700a7fc700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6ff4031090 0x7f6ff4033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f70000096a0 tx=0x7f7000006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:48.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.284+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6ffc01d7f0 con 0x7f700c0f52d0 2024-02-20T13:27:48.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.489+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f700c106ee0 con 0x7f700c0f52d0 2024-02-20T13:27:48.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.490+0000 7f6ff37fe700 1 -- 172.21.15.114:0/2175093882 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ffc0170c0 con 0x7f700c0f52d0 2024-02-20T13:27:48.491 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:48.492 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6ff4031090 msgr2=0x7f6ff4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6ff4031090 0x7f6ff4033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f70000096a0 tx=0x7f7000006b40).stop 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 msgr2=0x7f700c105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c105510 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6ffc0089f0 tx=0x7f6ffc00b040).stop 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 shutdown_connections 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6ff4031090 0x7f6ff4033550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 --2- 172.21.15.114:0/2175093882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f700c0f52d0 0x7f700c105510 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 >> 172.21.15.114:0/2175093882 conn(0x7f700c0f0a30 msgr2=0x7f700c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:48.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 shutdown_connections 2024-02-20T13:27:48.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:48.491+0000 7f700bfff700 1 -- 172.21.15.114:0/2175093882 wait complete. 2024-02-20T13:27:48.494 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:48.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:47.006543+0000 mgr.smithi038.powepi (mgr.14156) 326 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:49.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:48.491402+0000 mon.smithi038 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.114:0/2175093882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:50.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:50.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:50.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:50 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:49.006975+0000 mgr.smithi038.powepi (mgr.14156) 327 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 -- 172.21.15.114:0/3758867728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 msgr2=0x7f6f5c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 --2- 172.21.15.114:0/3758867728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c0f69b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6f44009fb0 tx=0x7f6f44009b30).stop 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 -- 172.21.15.114:0/3758867728 shutdown_connections 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 --2- 172.21.15.114:0/3758867728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c0f69b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 -- 172.21.15.114:0/3758867728 >> 172.21.15.114:0/3758867728 conn(0x7f6f5c0f1cf0 msgr2=0x7f6f5c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 -- 172.21.15.114:0/3758867728 shutdown_connections 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.373+0000 7f6f62345700 1 -- 172.21.15.114:0/3758867728 wait complete. 2024-02-20T13:27:52.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.374+0000 7f6f62345700 1 Processor -- start 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.374+0000 7f6f62345700 1 -- start start 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.374+0000 7f6f62345700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.374+0000 7f6f62345700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c106c70 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f5bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f5bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:40462/0 (socket says 172.21.15.114:40462) 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f5bfff700 1 -- 172.21.15.114:0/3466612781 learned_addr learned my addr 172.21.15.114:0/3466612781 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f5bfff700 1 -- 172.21.15.114:0/3466612781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44005040 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f5bfff700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c106780 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f6f440089f0 tx=0x7f6f4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f44009320 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c106e70 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6f44013ec0 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f44014d30 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.375+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c107300 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.376+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6f440095f0 con 0x7f6f5c0f6590 2024-02-20T13:27:52.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.376+0000 7f6f58ff9700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6f480310e0 0x7f6f480335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:52.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.376+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c045bb0 con 0x7f6f5c0f6590 2024-02-20T13:27:52.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.376+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6f440424a0 con 0x7f6f5c0f6590 2024-02-20T13:27:52.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.376+0000 7f6f5b7fe700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6f480310e0 0x7f6f480335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:52.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.377+0000 7f6f5b7fe700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6f480310e0 0x7f6f480335a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6f4c0096a0 tx=0x7f6f4c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:52.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.381+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6f4401d7f0 con 0x7f6f5c0f6590 2024-02-20T13:27:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.591+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f5c108100 con 0x7f6f5c0f6590 2024-02-20T13:27:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.592+0000 7f6f58ff9700 1 -- 172.21.15.114:0/3466612781 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f4401d7f0 con 0x7f6f5c0f6590 2024-02-20T13:27:52.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:52.594 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:52.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6f480310e0 msgr2=0x7f6f480335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:52.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6f480310e0 0x7f6f480335a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6f4c0096a0 tx=0x7f6f4c006b40).stop 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 msgr2=0x7f6f5c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c106780 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f6f440089f0 tx=0x7f6f4400b040).stop 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 shutdown_connections 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f6f480310e0 0x7f6f480335a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 --2- 172.21.15.114:0/3466612781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f5c0f6590 0x7f6f5c106780 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 >> 172.21.15.114:0/3466612781 conn(0x7f6f5c0f1cf0 msgr2=0x7f6f5c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 shutdown_connections 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:52.594+0000 7f6f62345700 1 -- 172.21.15.114:0/3466612781 wait complete. 2024-02-20T13:27:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:52.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:51.007408+0000 mgr.smithi038.powepi (mgr.14156) 328 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:53.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:52.593434+0000 mon.smithi038 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.114:0/3466612781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:54.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:54.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:54.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:54 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:53.007763+0000 mgr.smithi038.powepi (mgr.14156) 329 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:56.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 -- 172.21.15.114:0/729832209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 msgr2=0x7fb34c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:56.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 --2- 172.21.15.114:0/729832209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c0f58a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fb340009fb0 tx=0x7fb340009b30).stop 2024-02-20T13:27:56.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 -- 172.21.15.114:0/729832209 shutdown_connections 2024-02-20T13:27:56.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 --2- 172.21.15.114:0/729832209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c0f58a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:56.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 -- 172.21.15.114:0/729832209 >> 172.21.15.114:0/729832209 conn(0x7fb34c0f0be0 msgr2=0x7fb34c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:56.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 -- 172.21.15.114:0/729832209 shutdown_connections 2024-02-20T13:27:56.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 -- 172.21.15.114:0/729832209 wait complete. 2024-02-20T13:27:56.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.539+0000 7fb352656700 1 Processor -- start 2024-02-20T13:27:56.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb352656700 1 -- start start 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb352656700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb352656700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c105be0 con 0x7fb34c0f5480 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb34bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb34bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47126/0 (socket says 172.21.15.114:47126) 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb34bfff700 1 -- 172.21.15.114:0/3083096184 learned_addr learned my addr 172.21.15.114:0/3083096184 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.540+0000 7fb34bfff700 1 -- 172.21.15.114:0/3083096184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340005040 con 0x7fb34c0f5480 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb34bfff700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c1056f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fb3400089f0 tx=0x7fb34000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb340009320 con 0x7fb34c0f5480 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c105de0 con 0x7fb34c0f5480 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb340013ec0 con 0x7fb34c0f5480 2024-02-20T13:27:56.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c106270 con 0x7fb34c0f5480 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb340014d30 con 0x7fb34c0f5480 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.541+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb3400263a0 con 0x7fb34c0f5480 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.542+0000 7fb348ff9700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb334031090 0x7fb334033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.542+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c045bb0 con 0x7fb34c0f5480 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.542+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb340042630 con 0x7fb34c0f5480 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.542+0000 7fb34b7fe700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb334031090 0x7fb334033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:27:56.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.542+0000 7fb34b7fe700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb334031090 0x7fb334033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb33c0096a0 tx=0x7fb33c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:27:56.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.547+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb340012330 con 0x7fb34c0f5480 2024-02-20T13:27:56.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:55.008108+0000 mgr.smithi038.powepi (mgr.14156) 330 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:27:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.754+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb34c107080 con 0x7fb34c0f5480 2024-02-20T13:27:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.755+0000 7fb348ff9700 1 -- 172.21.15.114:0/3083096184 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb34001db70 con 0x7fb34c0f5480 2024-02-20T13:27:56.757 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:27:56.757 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb334031090 msgr2=0x7fb334033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb334031090 0x7fb334033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb33c0096a0 tx=0x7fb33c006b40).stop 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 msgr2=0x7fb34c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c1056f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fb3400089f0 tx=0x7fb34000b040).stop 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 shutdown_connections 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb334031090 0x7fb334033550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 --2- 172.21.15.114:0/3083096184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb34c0f5480 0x7fb34c1056f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 >> 172.21.15.114:0/3083096184 conn(0x7fb34c0f0be0 msgr2=0x7fb34c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:27:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 shutdown_connections 2024-02-20T13:27:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:27:56.756+0000 7fb352656700 1 -- 172.21.15.114:0/3083096184 wait complete. 2024-02-20T13:27:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:27:57.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:57 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:27:56.756477+0000 mon.smithi038 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.114:0/3083096184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:27:58.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:27:58.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:27:58.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:27:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:57.008474+0000 mgr.smithi038.powepi (mgr.14156) 331 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 -- 172.21.15.114:0/2569230865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 msgr2=0x7fb7340f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 --2- 172.21.15.114:0/2569230865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7340f69b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fb728004760 tx=0x7fb728009ad0).stop 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 -- 172.21.15.114:0/2569230865 shutdown_connections 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 --2- 172.21.15.114:0/2569230865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7340f69b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 -- 172.21.15.114:0/2569230865 >> 172.21.15.114:0/2569230865 conn(0x7fb7340f1cd0 msgr2=0x7fb7340f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 -- 172.21.15.114:0/2569230865 shutdown_connections 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 -- 172.21.15.114:0/2569230865 wait complete. 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.638+0000 7fb73a584700 1 Processor -- start 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb73a584700 1 -- start start 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb73a584700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7341067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb73a584700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb734106cb0 con 0x7fb7340f6590 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb733fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7341067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb733fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7341067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47140/0 (socket says 172.21.15.114:47140) 2024-02-20T13:28:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb733fff700 1 -- 172.21.15.114:0/553241949 learned_addr learned my addr 172.21.15.114:0/553241949 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb733fff700 1 -- 172.21.15.114:0/553241949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728009d70 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.639+0000 7fb733fff700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7341067c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fb728004170 tx=0x7fb728009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.640+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb72800e040 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.640+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb734106eb0 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.640+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb728008ef0 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.640+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb728014c90 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.640+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb734107340 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.640+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb728014420 con 0x7fb7340f6590 2024-02-20T13:28:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.641+0000 7fb730ff9700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb71c031090 0x7fb71c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.641+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb734045bb0 con 0x7fb7340f6590 2024-02-20T13:28:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.641+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb728042220 con 0x7fb7340f6590 2024-02-20T13:28:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.641+0000 7fb7337fe700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb71c031090 0x7fb71c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.641+0000 7fb7337fe700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb71c031090 0x7fb71c033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb7240096a0 tx=0x7fb724006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.646+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb728012d00 con 0x7fb7340f6590 2024-02-20T13:28:00.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:27:59.008872+0000 mgr.smithi038.powepi (mgr.14156) 332 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.855+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7341081f0 con 0x7fb7340f6590 2024-02-20T13:28:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.855+0000 7fb730ff9700 1 -- 172.21.15.114:0/553241949 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb728012b20 con 0x7fb7340f6590 2024-02-20T13:28:00.857 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:00.857 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb71c031090 msgr2=0x7fb71c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb71c031090 0x7fb71c033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb7240096a0 tx=0x7fb724006b40).stop 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 msgr2=0x7fb7341067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7341067c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fb728004170 tx=0x7fb728009ad0).stop 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 shutdown_connections 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fb71c031090 0x7fb71c033550 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 --2- 172.21.15.114:0/553241949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7340f6590 0x7fb7341067c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 >> 172.21.15.114:0/553241949 conn(0x7fb7340f1cd0 msgr2=0x7fb7340f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:00.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 shutdown_connections 2024-02-20T13:28:00.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:00.857+0000 7fb73a584700 1 -- 172.21.15.114:0/553241949 wait complete. 2024-02-20T13:28:00.860 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:01.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:01 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:00.857040+0000 mon.smithi038 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.114:0/553241949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:02.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:02.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:02.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:02 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:01.009188+0000 mgr.smithi038.powepi (mgr.14156) 333 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:04.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 -- 172.21.15.114:0/1158070513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 msgr2=0x7f93dc0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:04.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 --2- 172.21.15.114:0/1158070513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc0f4c60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f93cc009fb0 tx=0x7f93cc009b30).stop 2024-02-20T13:28:04.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 -- 172.21.15.114:0/1158070513 shutdown_connections 2024-02-20T13:28:04.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 --2- 172.21.15.114:0/1158070513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc0f4c60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:04.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 -- 172.21.15.114:0/1158070513 >> 172.21.15.114:0/1158070513 conn(0x7f93dc0f0360 msgr2=0x7f93dc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:04.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 -- 172.21.15.114:0/1158070513 shutdown_connections 2024-02-20T13:28:04.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.577+0000 7f93e1e09700 1 -- 172.21.15.114:0/1158070513 wait complete. 2024-02-20T13:28:04.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93e1e09700 1 Processor -- start 2024-02-20T13:28:04.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93e1e09700 1 -- start start 2024-02-20T13:28:04.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93e1e09700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93e1e09700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc106cb0 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93db7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93db7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:47150/0 (socket says 172.21.15.114:47150) 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93db7fe700 1 -- 172.21.15.114:0/3363604687 learned_addr learned my addr 172.21.15.114:0/3363604687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.578+0000 7f93db7fe700 1 -- 172.21.15.114:0/3363604687 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc005040 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93db7fe700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc1067c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f93cc0089f0 tx=0x7f93cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93cc009320 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc106eb0 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f93cc013ec0 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93cc014d30 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc107340 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.579+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f93cc014420 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.580+0000 7f93d3fff700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f93c8031090 0x7f93c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.580+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc045bb0 con 0x7f93dc0f4840 2024-02-20T13:28:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.580+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f93cc0424a0 con 0x7f93dc0f4840 2024-02-20T13:28:04.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.580+0000 7f93daffd700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f93c8031090 0x7f93c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:04.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.580+0000 7f93daffd700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f93c8031090 0x7f93c8033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f93c40096a0 tx=0x7f93c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:04.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.585+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f93cc014750 con 0x7f93dc0f4840 2024-02-20T13:28:04.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:03.009560+0000 mgr.smithi038.powepi (mgr.14156) 334 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:04.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.791+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93dc0fc070 con 0x7f93dc0f4840 2024-02-20T13:28:04.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.792+0000 7f93d3fff700 1 -- 172.21.15.114:0/3363604687 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93cc01d7f0 con 0x7f93dc0f4840 2024-02-20T13:28:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:04.793 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:04.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f93c8031090 msgr2=0x7f93c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:04.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f93c8031090 0x7f93c8033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f93c40096a0 tx=0x7f93c4006b40).stop 2024-02-20T13:28:04.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 msgr2=0x7f93dc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc1067c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f93cc0089f0 tx=0x7f93cc00b040).stop 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 shutdown_connections 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f93c8031090 0x7f93c8033550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 --2- 172.21.15.114:0/3363604687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93dc0f4840 0x7f93dc1067c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 >> 172.21.15.114:0/3363604687 conn(0x7f93dc0f0360 msgr2=0x7f93dc0f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 shutdown_connections 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:04.793+0000 7f93e1e09700 1 -- 172.21.15.114:0/3363604687 wait complete. 2024-02-20T13:28:04.795 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:05.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:05 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:04.792577+0000 mon.smithi038 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.114:0/3363604687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:06.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:06.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:06.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:06 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:05.009911+0000 mgr.smithi038.powepi (mgr.14156) 335 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:08.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- 172.21.15.114:0/3118401946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 msgr2=0x7f17b40f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 --2- 172.21.15.114:0/3118401946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b40f58d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f17a8009fb0 tx=0x7f17a8009b30).stop 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- 172.21.15.114:0/3118401946 shutdown_connections 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 --2- 172.21.15.114:0/3118401946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b40f58d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- 172.21.15.114:0/3118401946 >> 172.21.15.114:0/3118401946 conn(0x7f17b40f0c10 msgr2=0x7f17b40f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- 172.21.15.114:0/3118401946 shutdown_connections 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- 172.21.15.114:0/3118401946 wait complete. 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 Processor -- start 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- start start 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b41094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.535+0000 7f17ba22d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4060320 con 0x7f17b40f54b0 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17b37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b41094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17b37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b41094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41462/0 (socket says 172.21.15.114:41462) 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17b37fe700 1 -- 172.21.15.114:0/1304202632 learned_addr learned my addr 172.21.15.114:0/1304202632 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:08.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17b37fe700 1 -- 172.21.15.114:0/1304202632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a8005040 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17b37fe700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b41094a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f17a80089f0 tx=0x7f17a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17a8009320 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b4060520 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f17a8013ec0 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17a8014d30 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.536+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b4109b50 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.537+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f17a8014420 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.537+0000 7f179bfff700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f179c031090 0x7f179c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.537+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b4045bb0 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.537+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f17a8042400 con 0x7f17b40f54b0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.538+0000 7f17b2ffd700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f179c031090 0x7f179c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.538+0000 7f17b2ffd700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f179c031090 0x7f179c033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f17a40096a0 tx=0x7f17a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:08.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.543+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f17a8014750 con 0x7f17b40f54b0 2024-02-20T13:28:08.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:07.010243+0000 mgr.smithi038.powepi (mgr.14156) 336 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:08.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.750+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17b40fb090 con 0x7f17b40f54b0 2024-02-20T13:28:08.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.750+0000 7f179bfff700 1 -- 172.21.15.114:0/1304202632 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17a801d380 con 0x7f17b40f54b0 2024-02-20T13:28:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:08.751 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:08.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f179c031090 msgr2=0x7f179c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:08.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f179c031090 0x7f179c033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f17a40096a0 tx=0x7f17a4006b40).stop 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 msgr2=0x7f17b41094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b41094a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f17a80089f0 tx=0x7f17a800b040).stop 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 shutdown_connections 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f179c031090 0x7f179c033550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 --2- 172.21.15.114:0/1304202632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17b40f54b0 0x7f17b41094a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 >> 172.21.15.114:0/1304202632 conn(0x7f17b40f0c10 msgr2=0x7f17b40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 shutdown_connections 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:08.752+0000 7f17ba22d700 1 -- 172.21.15.114:0/1304202632 wait complete. 2024-02-20T13:28:08.753 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:09.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:09 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:08.750991+0000 mon.smithi038 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.114:0/1304202632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:10.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:10.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:10.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:09.010582+0000 mgr.smithi038.powepi (mgr.14156) 337 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:12.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- 172.21.15.114:0/3434414697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 msgr2=0x7fa4a40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:12.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 --2- 172.21.15.114:0/3434414697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a40f58a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fa494009fb0 tx=0x7fa494009b30).stop 2024-02-20T13:28:12.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- 172.21.15.114:0/3434414697 shutdown_connections 2024-02-20T13:28:12.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 --2- 172.21.15.114:0/3434414697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a40f58a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- 172.21.15.114:0/3434414697 >> 172.21.15.114:0/3434414697 conn(0x7fa4a40f0be0 msgr2=0x7fa4a40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- 172.21.15.114:0/3434414697 shutdown_connections 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- 172.21.15.114:0/3434414697 wait complete. 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 Processor -- start 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- start start 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a41067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.581+0000 7fa4a89f0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4106ce0 con 0x7fa4a40f5480 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a41067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a41067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:41490/0 (socket says 172.21.15.114:41490) 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a259c700 1 -- 172.21.15.114:0/738654033 learned_addr learned my addr 172.21.15.114:0/738654033 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a259c700 1 -- 172.21.15.114:0/738654033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494005040 con 0x7fa4a40f5480 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a259c700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a41067f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa4940089f0 tx=0x7fa49400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa494009320 con 0x7fa4a40f5480 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4106ee0 con 0x7fa4a40f5480 2024-02-20T13:28:12.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a4107370 con 0x7fa4a40f5480 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa494013ec0 con 0x7fa4a40f5480 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.582+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa494014d00 con 0x7fa4a40f5480 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.583+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa494014420 con 0x7fa4a40f5480 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.583+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4045bb0 con 0x7fa4a40f5480 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.583+0000 7fa492ffd700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa48c031090 0x7fa48c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.583+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa494042460 con 0x7fa4a40f5480 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.584+0000 7fa4a1d9b700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa48c031090 0x7fa48c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.584+0000 7fa4a1d9b700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa48c031090 0x7fa48c033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa4980096a0 tx=0x7fa498006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:12.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.589+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa494012d00 con 0x7fa4a40f5480 2024-02-20T13:28:12.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:11.010875+0000 mgr.smithi038.powepi (mgr.14156) 338 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:12.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.794+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4a40fc0a0 con 0x7fa4a40f5480 2024-02-20T13:28:12.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.795+0000 7fa492ffd700 1 -- 172.21.15.114:0/738654033 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa494012b20 con 0x7fa4a40f5480 2024-02-20T13:28:12.796 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:12.796 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:12.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa48c031090 msgr2=0x7fa48c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa48c031090 0x7fa48c033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa4980096a0 tx=0x7fa498006b40).stop 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 msgr2=0x7fa4a41067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a41067f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa4940089f0 tx=0x7fa49400b040).stop 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 shutdown_connections 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa48c031090 0x7fa48c033550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 --2- 172.21.15.114:0/738654033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4a40f5480 0x7fa4a41067f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 >> 172.21.15.114:0/738654033 conn(0x7fa4a40f0be0 msgr2=0x7fa4a40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 shutdown_connections 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:12.797+0000 7fa4a89f0700 1 -- 172.21.15.114:0/738654033 wait complete. 2024-02-20T13:28:12.798 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:13.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:13 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:12.795880+0000 mon.smithi038 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.114:0/738654033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:14.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:14.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:14.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:13.011172+0000 mgr.smithi038.powepi (mgr.14156) 339 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:16.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:15.011547+0000 mgr.smithi038.powepi (mgr.14156) 340 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 -- 172.21.15.114:0/3503873681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 msgr2=0x7f178c069fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 --2- 172.21.15.114:0/3503873681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c069fa0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1788004760 tx=0x7f1788009ad0).stop 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 -- 172.21.15.114:0/3503873681 shutdown_connections 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 --2- 172.21.15.114:0/3503873681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c069fa0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 -- 172.21.15.114:0/3503873681 >> 172.21.15.114:0/3503873681 conn(0x7f178c0f1cd0 msgr2=0x7f178c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 -- 172.21.15.114:0/3503873681 shutdown_connections 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.719+0000 7f1793456700 1 -- 172.21.15.114:0/3503873681 wait complete. 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f1793456700 1 Processor -- start 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f1793456700 1 -- start start 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f1793456700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c10ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:16.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f1793456700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c10d320 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f17911f2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c10ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f17911f2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c10ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55238/0 (socket says 172.21.15.114:55238) 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f17911f2700 1 -- 172.21.15.114:0/1326843173 learned_addr learned my addr 172.21.15.114:0/1326843173 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f17911f2700 1 -- 172.21.15.114:0/1326843173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1788009d70 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.720+0000 7f17911f2700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c10ce30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f1788013f90 tx=0x7f1788009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.721+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f178800e040 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.721+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1788009680 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.721+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c10d520 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.721+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1788014d20 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.721+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c10e130 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.722+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f178801f410 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.722+0000 7f1781ffb700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1778031090 0x7f1778033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.722+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1788042360 con 0x7f178c06baa0 2024-02-20T13:28:16.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.722+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c10d9b0 con 0x7f178c06baa0 2024-02-20T13:28:16.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.722+0000 7f17909f1700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1778031090 0x7f1778033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:16.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.722+0000 7f17909f1700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1778031090 0x7f1778033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f177c0096a0 tx=0x7f177c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:16.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.727+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f178c10d9b0 con 0x7f178c06baa0 2024-02-20T13:28:16.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.935+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f178c10d9b0 con 0x7f178c06baa0 2024-02-20T13:28:16.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.936+0000 7f1781ffb700 1 -- 172.21.15.114:0/1326843173 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1788012920 con 0x7f178c06baa0 2024-02-20T13:28:16.937 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:16.937 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1778031090 msgr2=0x7f1778033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1778031090 0x7f1778033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f177c0096a0 tx=0x7f177c006b40).stop 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 msgr2=0x7f178c10ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c10ce30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f1788013f90 tx=0x7f1788009ad0).stop 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 shutdown_connections 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1778031090 0x7f1778033550 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 --2- 172.21.15.114:0/1326843173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f178c06baa0 0x7f178c10ce30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 >> 172.21.15.114:0/1326843173 conn(0x7f178c0f1cd0 msgr2=0x7f178c0f2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 shutdown_connections 2024-02-20T13:28:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:16.938+0000 7f1793456700 1 -- 172.21.15.114:0/1326843173 wait complete. 2024-02-20T13:28:16.940 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:17.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:17 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:16.936889+0000 mon.smithi038 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.114:0/1326843173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:18.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:17.011867+0000 mgr.smithi038.powepi (mgr.14156) 341 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:18.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:18.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:20.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:19.012124+0000 mgr.smithi038.powepi (mgr.14156) 342 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:20.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 -- 172.21.15.114:0/1859514486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 msgr2=0x7f1c400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 --2- 172.21.15.114:0/1859514486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c400f58a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f1c30009fb0 tx=0x7f1c30009b30).stop 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 -- 172.21.15.114:0/1859514486 shutdown_connections 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 --2- 172.21.15.114:0/1859514486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c400f58a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 -- 172.21.15.114:0/1859514486 >> 172.21.15.114:0/1859514486 conn(0x7f1c400f0be0 msgr2=0x7f1c400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 -- 172.21.15.114:0/1859514486 shutdown_connections 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.814+0000 7f1c483d9700 1 -- 172.21.15.114:0/1859514486 wait complete. 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c483d9700 1 Processor -- start 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c483d9700 1 -- start start 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c483d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c40109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c483d9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40060300 con 0x7f1c400f5480 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c46175700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c40109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c46175700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c40109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55256/0 (socket says 172.21.15.114:55256) 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c46175700 1 -- 172.21.15.114:0/3836972750 learned_addr learned my addr 172.21.15.114:0/3836972750 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c46175700 1 -- 172.21.15.114:0/3836972750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30005040 con 0x7f1c400f5480 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.815+0000 7f1c46175700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c40109510 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1c300089f0 tx=0x7f1c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.816+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c30009320 con 0x7f1c400f5480 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.816+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c40060500 con 0x7f1c400f5480 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.816+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1c30013ec0 con 0x7f1c400f5480 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.816+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c30014d30 con 0x7f1c400f5480 2024-02-20T13:28:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.816+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c40109bc0 con 0x7f1c400f5480 2024-02-20T13:28:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.817+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1c300263a0 con 0x7f1c400f5480 2024-02-20T13:28:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.817+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c28004fa0 con 0x7f1c400f5480 2024-02-20T13:28:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.817+0000 7f1c36ffd700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c2c031090 0x7f1c2c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.817+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1c30042350 con 0x7f1c400f5480 2024-02-20T13:28:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.817+0000 7f1c45974700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c2c031090 0x7f1c2c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:20.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.817+0000 7f1c45974700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c2c031090 0x7f1c2c033550 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f1c3c0096a0 tx=0x7f1c3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:20.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:20.822+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1c300126f0 con 0x7f1c400f5480 2024-02-20T13:28:21.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.032+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c28005670 con 0x7f1c400f5480 2024-02-20T13:28:21.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.032+0000 7f1c36ffd700 1 -- 172.21.15.114:0/3836972750 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c30012330 con 0x7f1c400f5480 2024-02-20T13:28:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:21.033 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:21.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c2c031090 msgr2=0x7f1c2c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:21.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c2c031090 0x7f1c2c033550 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f1c3c0096a0 tx=0x7f1c3c006b40).stop 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 msgr2=0x7f1c40109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c40109510 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1c300089f0 tx=0x7f1c3000b040).stop 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 shutdown_connections 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1c2c031090 0x7f1c2c033550 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 --2- 172.21.15.114:0/3836972750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c400f5480 0x7f1c40109510 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 >> 172.21.15.114:0/3836972750 conn(0x7f1c400f0be0 msgr2=0x7f1c400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 shutdown_connections 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:21.034+0000 7f1c483d9700 1 -- 172.21.15.114:0/3836972750 wait complete. 2024-02-20T13:28:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:21.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:21 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:21.032951+0000 mon.smithi038 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.114:0/3836972750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:22.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:22.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:22.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:21.012369+0000 mgr.smithi038.powepi (mgr.14156) 343 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 -- 172.21.15.114:0/2809206572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 msgr2=0x7f004c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 --2- 172.21.15.114:0/2809206572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c0f56f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f0048009fb0 tx=0x7f0048009b30).stop 2024-02-20T13:28:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 -- 172.21.15.114:0/2809206572 shutdown_connections 2024-02-20T13:28:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 --2- 172.21.15.114:0/2809206572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c0f56f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 -- 172.21.15.114:0/2809206572 >> 172.21.15.114:0/2809206572 conn(0x7f004c0f0a30 msgr2=0x7f004c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 -- 172.21.15.114:0/2809206572 shutdown_connections 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 -- 172.21.15.114:0/2809206572 wait complete. 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 Processor -- start 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.494+0000 7f0053362700 1 -- start start 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0053362700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0053362700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004c106b30 con 0x7f004c0f52d0 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0052360700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0052360700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:55272/0 (socket says 172.21.15.114:55272) 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0052360700 1 -- 172.21.15.114:0/1319381692 learned_addr learned my addr 172.21.15.114:0/1319381692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0052360700 1 -- 172.21.15.114:0/1319381692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0048005040 con 0x7f004c0f52d0 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0052360700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c106640 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f00480089f0 tx=0x7f004800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048009320 con 0x7f004c0f52d0 2024-02-20T13:28:24.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004c106d30 con 0x7f004c0f52d0 2024-02-20T13:28:24.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0048013ec0 con 0x7f004c0f52d0 2024-02-20T13:28:24.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048014d30 con 0x7f004c0f52d0 2024-02-20T13:28:24.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.495+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004c1071c0 con 0x7f004c0f52d0 2024-02-20T13:28:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.496+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f00480095f0 con 0x7f004c0f52d0 2024-02-20T13:28:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.496+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004c045bb0 con 0x7f004c0f52d0 2024-02-20T13:28:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.496+0000 7f0042ffd700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0038031090 0x7f0038033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.496+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f00480424a0 con 0x7f004c0f52d0 2024-02-20T13:28:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.497+0000 7f0051b5f700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0038031090 0x7f0038033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.497+0000 7f0051b5f700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0038031090 0x7f0038033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f003c0096a0 tx=0x7f003c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:24.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.501+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f004801d980 con 0x7f004c0f52d0 2024-02-20T13:28:24.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:23.012671+0000 mgr.smithi038.powepi (mgr.14156) 344 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:24.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.707+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f004c108070 con 0x7f004c0f52d0 2024-02-20T13:28:24.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.708+0000 7f0042ffd700 1 -- 172.21.15.114:0/1319381692 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f004801d380 con 0x7f004c0f52d0 2024-02-20T13:28:24.708 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:24.709 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:24.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0038031090 msgr2=0x7f0038033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:24.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0038031090 0x7f0038033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f003c0096a0 tx=0x7f003c006b40).stop 2024-02-20T13:28:24.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 msgr2=0x7f004c106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:24.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c106640 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f00480089f0 tx=0x7f004800b040).stop 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 shutdown_connections 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0038031090 0x7f0038033550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 --2- 172.21.15.114:0/1319381692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f004c0f52d0 0x7f004c106640 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 >> 172.21.15.114:0/1319381692 conn(0x7f004c0f0a30 msgr2=0x7f004c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.709+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 shutdown_connections 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:24.710+0000 7f0053362700 1 -- 172.21.15.114:0/1319381692 wait complete. 2024-02-20T13:28:24.711 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:25.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:25 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:24.708469+0000 mon.smithi038 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.114:0/1319381692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:26.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:26.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:26.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:26 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:25.013065+0000 mgr.smithi038.powepi (mgr.14156) 345 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.488+0000 7f007df90700 1 -- 172.21.15.114:0/1460226931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 msgr2=0x7f00780f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.488+0000 7f007df90700 1 --2- 172.21.15.114:0/1460226931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00780f58a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0068009fb0 tx=0x7f0068009b30).stop 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.488+0000 7f007df90700 1 -- 172.21.15.114:0/1460226931 shutdown_connections 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.488+0000 7f007df90700 1 --2- 172.21.15.114:0/1460226931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00780f58a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.488+0000 7f007df90700 1 -- 172.21.15.114:0/1460226931 >> 172.21.15.114:0/1460226931 conn(0x7f00780f0be0 msgr2=0x7f00780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.489+0000 7f007df90700 1 -- 172.21.15.114:0/1460226931 shutdown_connections 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.489+0000 7f007df90700 1 -- 172.21.15.114:0/1460226931 wait complete. 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.489+0000 7f007df90700 1 Processor -- start 2024-02-20T13:28:28.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.489+0000 7f007df90700 1 -- start start 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.489+0000 7f007df90700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00781067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.489+0000 7f007df90700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078106ce0 con 0x7f00780f5480 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f00777fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00781067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f00777fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00781067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50390/0 (socket says 172.21.15.114:50390) 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f00777fe700 1 -- 172.21.15.114:0/2882008363 learned_addr learned my addr 172.21.15.114:0/2882008363 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f00777fe700 1 -- 172.21.15.114:0/2882008363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068005040 con 0x7f00780f5480 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f00777fe700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00781067f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f00680089f0 tx=0x7f006800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0068009320 con 0x7f00780f5480 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0078106ee0 con 0x7f00780f5480 2024-02-20T13:28:28.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0068013ec0 con 0x7f00780f5480 2024-02-20T13:28:28.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0068014d30 con 0x7f00780f5480 2024-02-20T13:28:28.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.490+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0078107370 con 0x7f00780f5480 2024-02-20T13:28:28.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.491+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f00680095f0 con 0x7f00780f5480 2024-02-20T13:28:28.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.491+0000 7f005ffff700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0060031090 0x7f0060033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:28.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.491+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f00680424e0 con 0x7f00780f5480 2024-02-20T13:28:28.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.491+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078045bb0 con 0x7f00780f5480 2024-02-20T13:28:28.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.492+0000 7f0076ffd700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0060031090 0x7f0060033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:28.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.492+0000 7f0076ffd700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0060031090 0x7f0060033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f006c0096a0 tx=0x7f006c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:28.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.496+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f006801d7f0 con 0x7f00780f5480 2024-02-20T13:28:28.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:27.013363+0000 mgr.smithi038.powepi (mgr.14156) 346 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:28.286575+0000 mon.smithi038 (mon.0) 315 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:28:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:28.286874+0000 mon.smithi038 (mon.0) 316 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:28:28.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.704+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00780fb060 con 0x7f00780f5480 2024-02-20T13:28:28.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.705+0000 7f005ffff700 1 -- 172.21.15.114:0/2882008363 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f006802b020 con 0x7f00780f5480 2024-02-20T13:28:28.707 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:28.707 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0060031090 msgr2=0x7f0060033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0060031090 0x7f0060033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f006c0096a0 tx=0x7f006c006b40).stop 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 msgr2=0x7f00781067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00781067f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f00680089f0 tx=0x7f006800b040).stop 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 shutdown_connections 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f0060031090 0x7f0060033550 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 --2- 172.21.15.114:0/2882008363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00780f5480 0x7f00781067f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 >> 172.21.15.114:0/2882008363 conn(0x7f00780f0be0 msgr2=0x7f00780f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 shutdown_connections 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:28.707+0000 7f007df90700 1 -- 172.21.15.114:0/2882008363 wait complete. 2024-02-20T13:28:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:28.706521+0000 mon.smithi038 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.114:0/2882008363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:30.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:30.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:30.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:30 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:29.013611+0000 mgr.smithi038.powepi (mgr.14156) 347 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:32.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 -- 172.21.15.114:0/296704120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 msgr2=0x7f9c800f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:32.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 --2- 172.21.15.114:0/296704120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c800f69b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f9c74004760 tx=0x7f9c74009ad0).stop 2024-02-20T13:28:32.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 -- 172.21.15.114:0/296704120 shutdown_connections 2024-02-20T13:28:32.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 --2- 172.21.15.114:0/296704120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c800f69b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:32.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 -- 172.21.15.114:0/296704120 >> 172.21.15.114:0/296704120 conn(0x7f9c800f1cd0 msgr2=0x7f9c800f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:32.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 -- 172.21.15.114:0/296704120 shutdown_connections 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.362+0000 7f9c8490a700 1 -- 172.21.15.114:0/296704120 wait complete. 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c8490a700 1 Processor -- start 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c8490a700 1 -- start start 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c8490a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c801067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c8490a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c80106cb0 con 0x7f9c800f6590 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c7e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c801067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c7e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c801067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:50416/0 (socket says 172.21.15.114:50416) 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c7e59c700 1 -- 172.21.15.114:0/931124185 learned_addr learned my addr 172.21.15.114:0/931124185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c7e59c700 1 -- 172.21.15.114:0/931124185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c74009d70 con 0x7f9c800f6590 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c7e59c700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c801067c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9c74013f10 tx=0x7f9c74009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c7400e040 con 0x7f9c800f6590 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c80106eb0 con 0x7f9c800f6590 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.363+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9c74007df0 con 0x7f9c800f6590 2024-02-20T13:28:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.364+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c74014d60 con 0x7f9c800f6590 2024-02-20T13:28:32.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.364+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c80107340 con 0x7f9c800f6590 2024-02-20T13:28:32.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.364+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9c74014520 con 0x7f9c800f6590 2024-02-20T13:28:32.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.364+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c80045bb0 con 0x7f9c800f6590 2024-02-20T13:28:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.364+0000 7f9c6effd700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c68031090 0x7f9c68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.364+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9c74042580 con 0x7f9c800f6590 2024-02-20T13:28:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.365+0000 7f9c7dd9b700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c68031090 0x7f9c68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.365+0000 7f9c7dd9b700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c68031090 0x7f9c68033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9c700096a0 tx=0x7f9c70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:32.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.369+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9c740283f0 con 0x7f9c800f6590 2024-02-20T13:28:32.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.575+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c80108170 con 0x7f9c800f6590 2024-02-20T13:28:32.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.576+0000 7f9c6effd700 1 -- 172.21.15.114:0/931124185 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c740262f0 con 0x7f9c800f6590 2024-02-20T13:28:32.578 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:32.578 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:32.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.577+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c68031090 msgr2=0x7f9c68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:32.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.577+0000 7f9c8490a700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c68031090 0x7f9c68033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9c700096a0 tx=0x7f9c70006b40).stop 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.577+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 msgr2=0x7f9c801067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.577+0000 7f9c8490a700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c801067c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9c74013f10 tx=0x7f9c74009ad0).stop 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.577+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 shutdown_connections 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.578+0000 7f9c8490a700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9c68031090 0x7f9c68033550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.578+0000 7f9c8490a700 1 --2- 172.21.15.114:0/931124185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c800f6590 0x7f9c801067c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.578+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 >> 172.21.15.114:0/931124185 conn(0x7f9c800f1cd0 msgr2=0x7f9c800f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.578+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 shutdown_connections 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:32.578+0000 7f9c8490a700 1 -- 172.21.15.114:0/931124185 wait complete. 2024-02-20T13:28:32.580 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:32.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:31.013991+0000 mgr.smithi038.powepi (mgr.14156) 348 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:33.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:33 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:32.577746+0000 mon.smithi038 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.114:0/931124185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:34.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:34.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:34.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:33.014250+0000 mgr.smithi038.powepi (mgr.14156) 349 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 -- 172.21.15.114:0/2876125696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 msgr2=0x7fa9100f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 --2- 172.21.15.114:0/2876125696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9100f69b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa900004760 tx=0x7fa900009ad0).stop 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 -- 172.21.15.114:0/2876125696 shutdown_connections 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 --2- 172.21.15.114:0/2876125696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9100f69b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 -- 172.21.15.114:0/2876125696 >> 172.21.15.114:0/2876125696 conn(0x7fa9100f1cd0 msgr2=0x7fa9100f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 -- 172.21.15.114:0/2876125696 shutdown_connections 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.412+0000 7fa914852700 1 -- 172.21.15.114:0/2876125696 wait complete. 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa914852700 1 Processor -- start 2024-02-20T13:28:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa914852700 1 -- start start 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa914852700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9101067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa914852700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa910106cb0 con 0x7fa9100f6590 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa90e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9101067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa90e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9101067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:52120/0 (socket says 172.21.15.114:52120) 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.413+0000 7fa90e59c700 1 -- 172.21.15.114:0/4054096189 learned_addr learned my addr 172.21.15.114:0/4054096189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa90e59c700 1 -- 172.21.15.114:0/4054096189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa900009d70 con 0x7fa9100f6590 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa90e59c700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9101067c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa900004170 tx=0x7fa900009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa90000e040 con 0x7fa9100f6590 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa910106eb0 con 0x7fa9100f6590 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa900008ef0 con 0x7fa9100f6590 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa900014c90 con 0x7fa9100f6590 2024-02-20T13:28:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.414+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa910107340 con 0x7fa9100f6590 2024-02-20T13:28:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.415+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa900014420 con 0x7fa9100f6590 2024-02-20T13:28:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.415+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa910045bb0 con 0x7fa9100f6590 2024-02-20T13:28:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.415+0000 7fa8feffd700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa8f8031090 0x7fa8f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.415+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa900042270 con 0x7fa9100f6590 2024-02-20T13:28:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.415+0000 7fa90dd9b700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa8f8031090 0x7fa8f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.415+0000 7fa90dd9b700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa8f8031090 0x7fa8f8033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa9040096a0 tx=0x7fa904006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:36.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.420+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa900026300 con 0x7fa9100f6590 2024-02-20T13:28:36.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.627+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9101081f0 con 0x7fa9100f6590 2024-02-20T13:28:36.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.628+0000 7fa8feffd700 1 -- 172.21.15.114:0/4054096189 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa900012b20 con 0x7fa9100f6590 2024-02-20T13:28:36.630 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:36.630 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:36.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa8f8031090 msgr2=0x7fa8f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:36.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa8f8031090 0x7fa8f8033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa9040096a0 tx=0x7fa904006b40).stop 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 msgr2=0x7fa9101067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9101067c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa900004170 tx=0x7fa900009ad0).stop 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 shutdown_connections 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fa8f8031090 0x7fa8f8033550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 --2- 172.21.15.114:0/4054096189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9100f6590 0x7fa9101067c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 >> 172.21.15.114:0/4054096189 conn(0x7fa9100f1cd0 msgr2=0x7fa9100f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 shutdown_connections 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:36.630+0000 7fa914852700 1 -- 172.21.15.114:0/4054096189 wait complete. 2024-02-20T13:28:36.632 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:36.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:35.014487+0000 mgr.smithi038.powepi (mgr.14156) 350 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:37.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:37 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:36.629850+0000 mon.smithi038 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.114:0/4054096189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:38.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:38.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:38.792 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:38 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:37.014851+0000 mgr.smithi038.powepi (mgr.14156) 351 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:40.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4281234636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 msgr2=0x7fbd880f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 --2- 172.21.15.114:0/4281234636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd880f58a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbd78009960 tx=0x7fbd78004d10).stop 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4281234636 shutdown_connections 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 --2- 172.21.15.114:0/4281234636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd880f58a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4281234636 >> 172.21.15.114:0/4281234636 conn(0x7fbd880f0be0 msgr2=0x7fbd880f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4281234636 shutdown_connections 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.147+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4281234636 wait complete. 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd8dcd9700 1 Processor -- start 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd8dcd9700 1 -- start start 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd8dcd9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd881094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:40.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd8dcd9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88060320 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd877fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd881094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd877fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd881094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:52134/0 (socket says 172.21.15.114:52134) 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.148+0000 7fbd877fe700 1 -- 172.21.15.114:0/4025617591 learned_addr learned my addr 172.21.15.114:0/4025617591 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd877fe700 1 -- 172.21.15.114:0/4025617591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd78009640 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd877fe700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd881094f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fbd78009610 tx=0x7fbd78004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd78018070 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88060520 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd7800b7e0 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7801d420 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.149+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88109ba0 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.150+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbd78024060 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.150+0000 7fbd7ffff700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbd74031090 0x7fbd74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.150+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88045bb0 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.150+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd78042650 con 0x7fbd880f5480 2024-02-20T13:28:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.150+0000 7fbd86ffd700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbd74031090 0x7fbd74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:40.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.151+0000 7fbd86ffd700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbd74031090 0x7fbd74033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbd700096a0 tx=0x7fbd70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:40.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.155+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd78011320 con 0x7fbd880f5480 2024-02-20T13:28:40.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.361+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd880fb060 con 0x7fbd880f5480 2024-02-20T13:28:40.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.361+0000 7fbd7ffff700 1 -- 172.21.15.114:0/4025617591 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd78026a40 con 0x7fbd880f5480 2024-02-20T13:28:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:40.364 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:40.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbd74031090 msgr2=0x7fbd74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:40.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbd74031090 0x7fbd74033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbd700096a0 tx=0x7fbd70006b40).stop 2024-02-20T13:28:40.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 msgr2=0x7fbd881094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd881094f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fbd78009610 tx=0x7fbd78004d10).stop 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 shutdown_connections 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fbd74031090 0x7fbd74033550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 --2- 172.21.15.114:0/4025617591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd880f5480 0x7fbd881094f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 >> 172.21.15.114:0/4025617591 conn(0x7fbd880f0be0 msgr2=0x7fbd880f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 shutdown_connections 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:40.363+0000 7fbd8dcd9700 1 -- 172.21.15.114:0/4025617591 wait complete. 2024-02-20T13:28:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:40.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:39.015267+0000 mgr.smithi038.powepi (mgr.14156) 352 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:40.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:40 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:40.363143+0000 mon.smithi038 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.114:0/4025617591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:41.946 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:41 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:41.015491+0000 mgr.smithi038.powepi (mgr.14156) 353 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:42.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:42.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:44.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 -- 172.21.15.114:0/390093008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 msgr2=0x7f32800f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:44.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 --2- 172.21.15.114:0/390093008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32800f56f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f3274009fb0 tx=0x7f3274009b30).stop 2024-02-20T13:28:44.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 -- 172.21.15.114:0/390093008 shutdown_connections 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 --2- 172.21.15.114:0/390093008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32800f56f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 -- 172.21.15.114:0/390093008 >> 172.21.15.114:0/390093008 conn(0x7f32800f0a30 msgr2=0x7f32800f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 -- 172.21.15.114:0/390093008 shutdown_connections 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 -- 172.21.15.114:0/390093008 wait complete. 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.277+0000 7f3285de7700 1 Processor -- start 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3285de7700 1 -- start start 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3285de7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32801065f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3285de7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3280106ae0 con 0x7f32800f52d0 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3284de5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32801065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3284de5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32801065f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:52150/0 (socket says 172.21.15.114:52150) 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3284de5700 1 -- 172.21.15.114:0/1040797351 learned_addr learned my addr 172.21.15.114:0/1040797351 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.278+0000 7f3284de5700 1 -- 172.21.15.114:0/1040797351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274005040 con 0x7f32800f52d0 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.279+0000 7f3284de5700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32801065f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f32740089f0 tx=0x7f327400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.279+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3274009320 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.279+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3280106ce0 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.279+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3274013ec0 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.279+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3274014d30 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.279+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3280107170 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.280+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3274014420 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.280+0000 7f327d7fa700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3270031090 0x7f3270033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.280+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f32740424c0 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.280+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3280045bb0 con 0x7f32800f52d0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.280+0000 7f327ffff700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3270031090 0x7f3270033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.280+0000 7f327ffff700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3270031090 0x7f3270033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f326c0096a0 tx=0x7f326c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.285+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3274014750 con 0x7f32800f52d0 2024-02-20T13:28:44.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.491+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3280107f70 con 0x7f32800f52d0 2024-02-20T13:28:44.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.492+0000 7f327d7fa700 1 -- 172.21.15.114:0/1040797351 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f327401d380 con 0x7f32800f52d0 2024-02-20T13:28:44.493 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:44.494 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:44.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3270031090 msgr2=0x7f3270033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:44.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3270031090 0x7f3270033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f326c0096a0 tx=0x7f326c006b40).stop 2024-02-20T13:28:44.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 msgr2=0x7f32801065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:44.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32801065f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f32740089f0 tx=0x7f327400b040).stop 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 shutdown_connections 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3270031090 0x7f3270033550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 --2- 172.21.15.114:0/1040797351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32800f52d0 0x7f32801065f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 >> 172.21.15.114:0/1040797351 conn(0x7f32800f0a30 msgr2=0x7f32800f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 shutdown_connections 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:44.493+0000 7f3285de7700 1 -- 172.21.15.114:0/1040797351 wait complete. 2024-02-20T13:28:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:44.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:44 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:43.015887+0000 mgr.smithi038.powepi (mgr.14156) 354 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:45.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:45 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:44.493417+0000 mon.smithi038 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.114:0/1040797351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:46.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:46.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:46.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:46 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:45.016319+0000 mgr.smithi038.powepi (mgr.14156) 355 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- 172.21.15.114:0/430025632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 msgr2=0x7ff7c80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 --2- 172.21.15.114:0/430025632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c80f56f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ff7b8009fb0 tx=0x7ff7b8009b30).stop 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- 172.21.15.114:0/430025632 shutdown_connections 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 --2- 172.21.15.114:0/430025632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c80f56f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- 172.21.15.114:0/430025632 >> 172.21.15.114:0/430025632 conn(0x7ff7c80f0a30 msgr2=0x7ff7c80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- 172.21.15.114:0/430025632 shutdown_connections 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- 172.21.15.114:0/430025632 wait complete. 2024-02-20T13:28:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 Processor -- start 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- start start 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c8109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7cf095700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8060320 con 0x7ff7c80f52d0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.172+0000 7ff7ce093700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c8109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7ce093700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c8109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35348/0 (socket says 172.21.15.114:35348) 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7ce093700 1 -- 172.21.15.114:0/1154410221 learned_addr learned my addr 172.21.15.114:0/1154410221 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7ce093700 1 -- 172.21.15.114:0/1154410221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8005040 con 0x7ff7c80f52d0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7ce093700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c8109310 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7ff7b80089f0 tx=0x7ff7b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7b8009320 con 0x7ff7c80f52d0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c8060520 con 0x7ff7c80f52d0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff7b8013ec0 con 0x7ff7c80f52d0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7b8014d30 con 0x7ff7c80f52d0 2024-02-20T13:28:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.173+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c81099c0 con 0x7ff7c80f52d0 2024-02-20T13:28:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.174+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff7b80095f0 con 0x7ff7c80f52d0 2024-02-20T13:28:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.174+0000 7ff7beffd700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7b4031090 0x7ff7b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.174+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff7b80423e0 con 0x7ff7c80f52d0 2024-02-20T13:28:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.174+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c8045bb0 con 0x7ff7c80f52d0 2024-02-20T13:28:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.174+0000 7ff7cd892700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7b4031090 0x7ff7b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.175+0000 7ff7cd892700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7b4031090 0x7ff7b4033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff7c40096a0 tx=0x7ff7c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.179+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff7b801dd70 con 0x7ff7c80f52d0 2024-02-20T13:28:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.384+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7c810a1b0 con 0x7ff7c80f52d0 2024-02-20T13:28:48.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.384+0000 7ff7beffd700 1 -- 172.21.15.114:0/1154410221 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7b802b020 con 0x7ff7c80f52d0 2024-02-20T13:28:48.386 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:48.387 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7b4031090 msgr2=0x7ff7b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7b4031090 0x7ff7b4033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff7c40096a0 tx=0x7ff7c4006b40).stop 2024-02-20T13:28:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 msgr2=0x7ff7c8109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c8109310 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7ff7b80089f0 tx=0x7ff7b800b040).stop 2024-02-20T13:28:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 shutdown_connections 2024-02-20T13:28:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff7b4031090 0x7ff7b4033550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 --2- 172.21.15.114:0/1154410221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c80f52d0 0x7ff7c8109310 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 >> 172.21.15.114:0/1154410221 conn(0x7ff7c80f0a30 msgr2=0x7ff7c80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 shutdown_connections 2024-02-20T13:28:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:48.386+0000 7ff7cf095700 1 -- 172.21.15.114:0/1154410221 wait complete. 2024-02-20T13:28:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:48.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:48 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:47.016581+0000 mgr.smithi038.powepi (mgr.14156) 356 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:49.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:49 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:48.386038+0000 mon.smithi038 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.114:0/1154410221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:50.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:50.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:50.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:50 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:49.016950+0000 mgr.smithi038.powepi (mgr.14156) 357 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 -- 172.21.15.114:0/695486539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 msgr2=0x7f76f40f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 --2- 172.21.15.114:0/695486539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f40f58d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f76e8009fb0 tx=0x7f76e8009b30).stop 2024-02-20T13:28:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 -- 172.21.15.114:0/695486539 shutdown_connections 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 --2- 172.21.15.114:0/695486539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f40f58d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 -- 172.21.15.114:0/695486539 >> 172.21.15.114:0/695486539 conn(0x7f76f40f0c10 msgr2=0x7f76f40f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 -- 172.21.15.114:0/695486539 shutdown_connections 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 -- 172.21.15.114:0/695486539 wait complete. 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.112+0000 7f76fc1fa700 1 Processor -- start 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76fc1fa700 1 -- start start 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76fc1fa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f4109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76fc1fa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4060300 con 0x7f76f40f54b0 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76f9f96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f4109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76f9f96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f4109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:35374/0 (socket says 172.21.15.114:35374) 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76f9f96700 1 -- 172.21.15.114:0/1824870644 learned_addr learned my addr 172.21.15.114:0/1824870644 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76f9f96700 1 -- 172.21.15.114:0/1824870644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8005040 con 0x7f76f40f54b0 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76f9f96700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f4109540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f76e8000c00 tx=0x7f76e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:52.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.113+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e80095f0 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f4060500 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f76e8007e50 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e8014e70 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f4109bf0 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f76e80079c0 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f40591e0 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.114+0000 7f76e67fc700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f76e0031090 0x7f76e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.115+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f76e8041c50 con 0x7f76f40f54b0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.115+0000 7f76f9795700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f76e0031090 0x7f76e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.115+0000 7f76f9795700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f76e0031090 0x7f76e0033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f76f00096a0 tx=0x7f76f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.119+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f76e801b910 con 0x7f76f40f54b0 2024-02-20T13:28:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.329+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76f4045bb0 con 0x7f76f40f54b0 2024-02-20T13:28:52.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.329+0000 7f76e67fc700 1 -- 172.21.15.114:0/1824870644 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f76e801b2c0 con 0x7f76f40f54b0 2024-02-20T13:28:52.331 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:52.332 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f76e0031090 msgr2=0x7f76e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f76e0031090 0x7f76e0033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f76f00096a0 tx=0x7f76f0006b40).stop 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 msgr2=0x7f76f4109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f4109540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f76e8000c00 tx=0x7f76e800b040).stop 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 shutdown_connections 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f76e0031090 0x7f76e0033550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 --2- 172.21.15.114:0/1824870644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f40f54b0 0x7f76f4109540 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 >> 172.21.15.114:0/1824870644 conn(0x7f76f40f0c10 msgr2=0x7f76f40f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 shutdown_connections 2024-02-20T13:28:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:52.331+0000 7f76fc1fa700 1 -- 172.21.15.114:0/1824870644 wait complete. 2024-02-20T13:28:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:52.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:52 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:51.017337+0000 mgr.smithi038.powepi (mgr.14156) 358 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:53.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:53 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:52.330159+0000 mon.smithi038 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.114:0/1824870644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:54.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:54.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:54.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:54 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:53.017753+0000 mgr.smithi038.powepi (mgr.14156) 359 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:55.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.916+0000 7ff01c51c700 1 -- 172.21.15.114:0/1267268294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 msgr2=0x7ff0140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:55.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.916+0000 7ff01c51c700 1 --2- 172.21.15.114:0/1267268294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0140f58a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff010009fb0 tx=0x7ff010009b30).stop 2024-02-20T13:28:55.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 -- 172.21.15.114:0/1267268294 shutdown_connections 2024-02-20T13:28:55.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 --2- 172.21.15.114:0/1267268294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0140f58a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:55.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 -- 172.21.15.114:0/1267268294 >> 172.21.15.114:0/1267268294 conn(0x7ff0140f0be0 msgr2=0x7ff0140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:55.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 -- 172.21.15.114:0/1267268294 shutdown_connections 2024-02-20T13:28:55.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 -- 172.21.15.114:0/1267268294 wait complete. 2024-02-20T13:28:55.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 Processor -- start 2024-02-20T13:28:55.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 -- start start 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0141094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.917+0000 7ff01c51c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff014060320 con 0x7ff0140f5480 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.918+0000 7ff01a2b8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0141094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.918+0000 7ff01a2b8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0141094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43882/0 (socket says 172.21.15.114:43882) 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.918+0000 7ff01a2b8700 1 -- 172.21.15.114:0/4271083401 learned_addr learned my addr 172.21.15.114:0/4271083401 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.918+0000 7ff01a2b8700 1 -- 172.21.15.114:0/4271083401 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff010005040 con 0x7ff0140f5480 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.918+0000 7ff01a2b8700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0141094f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7ff0100089f0 tx=0x7ff01000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff010009320 con 0x7ff0140f5480 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff014060520 con 0x7ff0140f5480 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff010013ec0 con 0x7ff0140f5480 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff010014d30 con 0x7ff0140f5480 2024-02-20T13:28:55.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff014109ba0 con 0x7ff0140f5480 2024-02-20T13:28:55.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff0100095f0 con 0x7ff0140f5480 2024-02-20T13:28:55.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.919+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff014045bb0 con 0x7ff0140f5480 2024-02-20T13:28:55.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.920+0000 7ff00affd700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff000031090 0x7ff000033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:55.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.920+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff0100424a0 con 0x7ff0140f5480 2024-02-20T13:28:55.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.920+0000 7ff019ab7700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff000031090 0x7ff000033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:55.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.920+0000 7ff019ab7700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff000031090 0x7ff000033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff0040096a0 tx=0x7ff004006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:55.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:55.925+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff01001d7f0 con 0x7ff0140f5480 2024-02-20T13:28:56.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.128+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0140faf70 con 0x7ff0140f5480 2024-02-20T13:28:56.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.129+0000 7ff00affd700 1 -- 172.21.15.114:0/4271083401 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff01001d7f0 con 0x7ff0140f5480 2024-02-20T13:28:56.129 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:28:56.130 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff000031090 msgr2=0x7ff000033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff000031090 0x7ff000033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff0040096a0 tx=0x7ff004006b40).stop 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 msgr2=0x7ff0141094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0141094f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7ff0100089f0 tx=0x7ff01000b040).stop 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 shutdown_connections 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7ff000031090 0x7ff000033550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 --2- 172.21.15.114:0/4271083401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0140f5480 0x7ff0141094f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 >> 172.21.15.114:0/4271083401 conn(0x7ff0140f0be0 msgr2=0x7ff0140f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 shutdown_connections 2024-02-20T13:28:56.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:56.130+0000 7ff01c51c700 1 -- 172.21.15.114:0/4271083401 wait complete. 2024-02-20T13:28:56.132 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:28:56.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:55.018158+0000 mgr.smithi038.powepi (mgr.14156) 360 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:56.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:56 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:28:56.129436+0000 mon.smithi038 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.114:0/4271083401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:28:57.849 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:28:57.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:28:58.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:28:58 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:57.018580+0000 mgr.smithi038.powepi (mgr.14156) 361 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.881+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1180244912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 msgr2=0x7f9a980f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.881+0000 7f9a9e8ca700 1 --2- 172.21.15.114:0/1180244912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a980f6800 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f9a88009fb0 tx=0x7f9a88009b30).stop 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.881+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1180244912 shutdown_connections 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.881+0000 7f9a9e8ca700 1 --2- 172.21.15.114:0/1180244912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a980f6800 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.881+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1180244912 >> 172.21.15.114:0/1180244912 conn(0x7f9a980f1b40 msgr2=0x7f9a980f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.882+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1180244912 shutdown_connections 2024-02-20T13:28:59.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.882+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1180244912 wait complete. 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.882+0000 7f9a9e8ca700 1 Processor -- start 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.882+0000 7f9a9e8ca700 1 -- start start 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.882+0000 7f9a9e8ca700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a98106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.882+0000 7f9a9e8ca700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98106b10 con 0x7f9a980f63e0 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9d8c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a98106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9d8c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a98106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43890/0 (socket says 172.21.15.114:43890) 2024-02-20T13:28:59.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9d8c8700 1 -- 172.21.15.114:0/1576237262 learned_addr learned my addr 172.21.15.114:0/1576237262 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9d8c8700 1 -- 172.21.15.114:0/1576237262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88005040 con 0x7f9a980f63e0 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9d8c8700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a98106620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f9a880089f0 tx=0x7f9a8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a88009320 con 0x7f9a980f63e0 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a98106d10 con 0x7f9a980f63e0 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9a88013ec0 con 0x7f9a980f63e0 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a88014d30 con 0x7f9a980f63e0 2024-02-20T13:28:59.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.883+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a981071a0 con 0x7f9a980f63e0 2024-02-20T13:28:59.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.884+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9a880095f0 con 0x7f9a980f63e0 2024-02-20T13:28:59.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.884+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a98045bb0 con 0x7f9a980f63e0 2024-02-20T13:28:59.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.884+0000 7f9a8e7fc700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9a840310e0 0x7f9a840335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:28:59.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.884+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9a880424a0 con 0x7f9a980f63e0 2024-02-20T13:28:59.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.884+0000 7f9a9d0c7700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9a840310e0 0x7f9a840335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:28:59.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.885+0000 7f9a9d0c7700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9a840310e0 0x7f9a840335a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9a940096a0 tx=0x7f9a94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:28:59.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:28:59.889+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9a8801d7f0 con 0x7f9a980f63e0 2024-02-20T13:29:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.096+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a98108050 con 0x7f9a980f63e0 2024-02-20T13:29:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.096+0000 7f9a8e7fc700 1 -- 172.21.15.114:0/1576237262 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a8801d7f0 con 0x7f9a980f63e0 2024-02-20T13:29:00.097 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:00.097 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9a840310e0 msgr2=0x7f9a840335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9a840310e0 0x7f9a840335a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9a940096a0 tx=0x7f9a94006b40).stop 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 msgr2=0x7f9a98106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a98106620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f9a880089f0 tx=0x7f9a8800b040).stop 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 shutdown_connections 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f9a840310e0 0x7f9a840335a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 --2- 172.21.15.114:0/1576237262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a980f63e0 0x7f9a98106620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 >> 172.21.15.114:0/1576237262 conn(0x7f9a980f1b40 msgr2=0x7f9a980f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 shutdown_connections 2024-02-20T13:29:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:00.098+0000 7f9a9e8ca700 1 -- 172.21.15.114:0/1576237262 wait complete. 2024-02-20T13:29:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:00.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:28:59.019016+0000 mgr.smithi038.powepi (mgr.14156) 362 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:00.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:00 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:00.097051+0000 mon.smithi038 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.114:0/1576237262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:01.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:01.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:02.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:02 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:01.019500+0000 mgr.smithi038.powepi (mgr.14156) 363 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 -- 172.21.15.114:0/986420240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d300f6590 msgr2=0x7f3d300f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 --2- 172.21.15.114:0/986420240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d300f6590 0x7f3d300f69b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f3d20004760 tx=0x7f3d20009ad0).stop 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 -- 172.21.15.114:0/986420240 shutdown_connections 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 --2- 172.21.15.114:0/986420240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d300f6590 0x7f3d300f69b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 -- 172.21.15.114:0/986420240 >> 172.21.15.114:0/986420240 conn(0x7f3d300f1cd0 msgr2=0x7f3d300f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 -- 172.21.15.114:0/986420240 shutdown_connections 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.741+0000 7f3d37d3b700 1 -- 172.21.15.114:0/986420240 wait complete. 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d37d3b700 1 Processor -- start 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d37d3b700 1 -- start start 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d37d3b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 0x7f3d30069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d37d3b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3006be60 con 0x7f3d3006ba40 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d35ad7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 0x7f3d30069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d35ad7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 0x7f3d30069f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:43910/0 (socket says 172.21.15.114:43910) 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.742+0000 7f3d35ad7700 1 -- 172.21.15.114:0/4146409051 learned_addr learned my addr 172.21.15.114:0/4146409051 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d35ad7700 1 -- 172.21.15.114:0/4146409051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20009d70 con 0x7f3d3006ba40 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d35ad7700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 0x7f3d30069f00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3d20009820 tx=0x7f3d20009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2000e040 con 0x7f3d3006ba40 2024-02-20T13:29:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3006a3f0 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3d200095b0 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d20014be0 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.743+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3006a8e0 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.744+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3d20014420 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.744+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d30045bb0 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.744+0000 7f3d267fc700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d1c031090 0x7f3d1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.744+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3d20042150 con 0x7f3d3006ba40 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.744+0000 7f3d352d6700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d1c031090 0x7f3d1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.745+0000 7f3d352d6700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d1c031090 0x7f3d1c033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3d2c0096a0 tx=0x7f3d2c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.749+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3d200262b0 con 0x7f3d3006ba40 2024-02-20T13:29:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.955+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d3010caf0 con 0x7f3d3006ba40 2024-02-20T13:29:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.956+0000 7f3d267fc700 1 -- 172.21.15.114:0/4146409051 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d2001eb60 con 0x7f3d3006ba40 2024-02-20T13:29:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:03.957 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d1c031090 msgr2=0x7f3d1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d1c031090 0x7f3d1c033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3d2c0096a0 tx=0x7f3d2c006b40).stop 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 msgr2=0x7f3d30069f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 0x7f3d30069f00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3d20009820 tx=0x7f3d20009ad0).stop 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 shutdown_connections 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d1c031090 0x7f3d1c033550 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 --2- 172.21.15.114:0/4146409051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3006ba40 0x7f3d30069f00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 >> 172.21.15.114:0/4146409051 conn(0x7f3d300f1cd0 msgr2=0x7f3d300f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 shutdown_connections 2024-02-20T13:29:03.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:03.958+0000 7f3d37d3b700 1 -- 172.21.15.114:0/4146409051 wait complete. 2024-02-20T13:29:03.960 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:04.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:03.019886+0000 mgr.smithi038.powepi (mgr.14156) 364 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:04.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:04 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:03.956780+0000 mon.smithi038 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.114:0/4146409051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:05.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:05.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:06.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:06 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:05.020252+0000 mgr.smithi038.powepi (mgr.14156) 365 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3323249324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 msgr2=0x7f1fc00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 --2- 172.21.15.114:0/3323249324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00f58d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f1fbc009fb0 tx=0x7f1fbc009b30).stop 2024-02-20T13:29:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3323249324 shutdown_connections 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 --2- 172.21.15.114:0/3323249324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00f58d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3323249324 >> 172.21.15.114:0/3323249324 conn(0x7f1fc00f0c10 msgr2=0x7f1fc00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3323249324 shutdown_connections 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3323249324 wait complete. 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.574+0000 7f1fc7c6a700 1 Processor -- start 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc7c6a700 1 -- start start 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc7c6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc0109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc7c6a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0060300 con 0x7f1fc00f54b0 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc5a06700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc0109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc5a06700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc0109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44754/0 (socket says 172.21.15.114:44754) 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc5a06700 1 -- 172.21.15.114:0/3997254210 learned_addr learned my addr 172.21.15.114:0/3997254210 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc5a06700 1 -- 172.21.15.114:0/3997254210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc005040 con 0x7f1fc00f54b0 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc5a06700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc0109540 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f1fbc0089f0 tx=0x7f1fbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fbc009320 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc0060500 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.575+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1fbc013ec0 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.576+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fbc014d30 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.576+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc0109bf0 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.576+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1fbc0095f0 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.576+0000 7f1fb67fc700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fac031090 0x7f1fac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.576+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1fbc0424a0 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.576+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc00591e0 con 0x7f1fc00f54b0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.577+0000 7f1fc5205700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fac031090 0x7f1fac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.577+0000 7f1fc5205700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fac031090 0x7f1fac033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1fb00096a0 tx=0x7f1fb0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:07.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.581+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1fbc01dd70 con 0x7f1fc00f54b0 2024-02-20T13:29:07.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.788+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc0045bb0 con 0x7f1fc00f54b0 2024-02-20T13:29:07.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.789+0000 7f1fb67fc700 1 -- 172.21.15.114:0/3997254210 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fbc02b020 con 0x7f1fc00f54b0 2024-02-20T13:29:07.790 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:07.790 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fac031090 msgr2=0x7f1fac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fac031090 0x7f1fac033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1fb00096a0 tx=0x7f1fb0006b40).stop 2024-02-20T13:29:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 msgr2=0x7f1fc0109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc0109540 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f1fbc0089f0 tx=0x7f1fbc00b040).stop 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 shutdown_connections 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f1fac031090 0x7f1fac033550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 --2- 172.21.15.114:0/3997254210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fc00f54b0 0x7f1fc0109540 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.790+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 >> 172.21.15.114:0/3997254210 conn(0x7f1fc00f0c10 msgr2=0x7f1fc00f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.791+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 shutdown_connections 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:07.791+0000 7f1fc7c6a700 1 -- 172.21.15.114:0/3997254210 wait complete. 2024-02-20T13:29:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:08.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:07.020673+0000 mgr.smithi038.powepi (mgr.14156) 366 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:08.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:08 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:07.789589+0000 mon.smithi038 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.114:0/3997254210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:09.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:09.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:10.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:10 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:09.020954+0000 mgr.smithi038.powepi (mgr.14156) 367 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- 172.21.15.114:0/705918739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 msgr2=0x7fad1c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 --2- 172.21.15.114:0/705918739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c0f69b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fad0c004760 tx=0x7fad0c009ad0).stop 2024-02-20T13:29:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- 172.21.15.114:0/705918739 shutdown_connections 2024-02-20T13:29:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 --2- 172.21.15.114:0/705918739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c0f69b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- 172.21.15.114:0/705918739 >> 172.21.15.114:0/705918739 conn(0x7fad1c0f1cd0 msgr2=0x7fad1c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- 172.21.15.114:0/705918739 shutdown_connections 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- 172.21.15.114:0/705918739 wait complete. 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 Processor -- start 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- start start 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.467+0000 7fad24612700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c106cb0 con 0x7fad1c0f6590 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad223ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad223ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:44772/0 (socket says 172.21.15.114:44772) 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad223ae700 1 -- 172.21.15.114:0/3537189171 learned_addr learned my addr 172.21.15.114:0/3537189171 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad223ae700 1 -- 172.21.15.114:0/3537189171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c009d70 con 0x7fad1c0f6590 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad223ae700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c1067c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fad0c013f10 tx=0x7fad0c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad0c00e040 con 0x7fad1c0f6590 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c106eb0 con 0x7fad1c0f6590 2024-02-20T13:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad0c007df0 con 0x7fad1c0f6590 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad0c014d60 con 0x7fad1c0f6590 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.468+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c107340 con 0x7fad1c0f6590 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.469+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fad0c01f460 con 0x7fad1c0f6590 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.469+0000 7fad12ffd700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad08031090 0x7fad08033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.469+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fad0c042500 con 0x7fad1c0f6590 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.469+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c045bb0 con 0x7fad1c0f6590 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.470+0000 7fad21bad700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad08031090 0x7fad08033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.470+0000 7fad21bad700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad08031090 0x7fad08033550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fad180096a0 tx=0x7fad18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.474+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad0c0283f0 con 0x7fad1c0f6590 2024-02-20T13:29:11.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.680+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad1c107fc0 con 0x7fad1c0f6590 2024-02-20T13:29:11.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.681+0000 7fad12ffd700 1 -- 172.21.15.114:0/3537189171 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad0c012330 con 0x7fad1c0f6590 2024-02-20T13:29:11.682 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:11.682 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:11.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad08031090 msgr2=0x7fad08033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad08031090 0x7fad08033550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fad180096a0 tx=0x7fad18006b40).stop 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 msgr2=0x7fad1c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c1067c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fad0c013f10 tx=0x7fad0c009ad0).stop 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 shutdown_connections 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7fad08031090 0x7fad08033550 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 --2- 172.21.15.114:0/3537189171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad1c0f6590 0x7fad1c1067c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 >> 172.21.15.114:0/3537189171 conn(0x7fad1c0f1cd0 msgr2=0x7fad1c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 shutdown_connections 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:11.683+0000 7fad24612700 1 -- 172.21.15.114:0/3537189171 wait complete. 2024-02-20T13:29:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:12.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:11.021351+0000 mgr.smithi038.powepi (mgr.14156) 368 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:12.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:12 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:11.681921+0000 mon.smithi038 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.114:0/3537189171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:13.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:13.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:14.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:14 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:13.021693+0000 mgr.smithi038.powepi (mgr.14156) 369 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 -- 172.21.15.114:0/3464137954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 msgr2=0x7f350c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 --2- 172.21.15.114:0/3464137954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c0f69b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f3500009fb0 tx=0x7f3500009b30).stop 2024-02-20T13:29:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 -- 172.21.15.114:0/3464137954 shutdown_connections 2024-02-20T13:29:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 --2- 172.21.15.114:0/3464137954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c0f69b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 -- 172.21.15.114:0/3464137954 >> 172.21.15.114:0/3464137954 conn(0x7f350c0f1cf0 msgr2=0x7f350c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 -- 172.21.15.114:0/3464137954 shutdown_connections 2024-02-20T13:29:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.407+0000 7f3510a14700 1 -- 172.21.15.114:0/3464137954 wait complete. 2024-02-20T13:29:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f3510a14700 1 Processor -- start 2024-02-20T13:29:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f3510a14700 1 -- start start 2024-02-20T13:29:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f3510a14700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f3510a14700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c106cf0 con 0x7f350c0f6590 2024-02-20T13:29:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f350a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f350a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:33026/0 (socket says 172.21.15.114:33026) 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.408+0000 7f350a59c700 1 -- 172.21.15.114:0/1274310850 learned_addr learned my addr 172.21.15.114:0/1274310850 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f350a59c700 1 -- 172.21.15.114:0/1274310850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500005040 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f350a59c700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c106800 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f35000089f0 tx=0x7f350000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3500009320 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3500013ec0 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c106ef0 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3500014d30 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.409+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c107380 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.410+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f35000095f0 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.410+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c045bb0 con 0x7f350c0f6590 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.410+0000 7f34faffd700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34f4031090 0x7f34f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.410+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f35000424e0 con 0x7f350c0f6590 2024-02-20T13:29:15.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.410+0000 7f3509d9b700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34f4031090 0x7f34f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:15.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.411+0000 7f3509d9b700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34f4031090 0x7f34f4033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f34fc0096a0 tx=0x7f34fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:15.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.415+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f350001d7f0 con 0x7f350c0f6590 2024-02-20T13:29:15.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.618+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f350c0fc080 con 0x7f350c0f6590 2024-02-20T13:29:15.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.619+0000 7f34faffd700 1 -- 172.21.15.114:0/1274310850 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f350000fe40 con 0x7f350c0f6590 2024-02-20T13:29:15.621 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:15.621 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34f4031090 msgr2=0x7f34f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34f4031090 0x7f34f4033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f34fc0096a0 tx=0x7f34fc006b40).stop 2024-02-20T13:29:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 msgr2=0x7f350c106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c106800 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f35000089f0 tx=0x7f350000b040).stop 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 shutdown_connections 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f34f4031090 0x7f34f4033550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 --2- 172.21.15.114:0/1274310850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f350c0f6590 0x7f350c106800 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 >> 172.21.15.114:0/1274310850 conn(0x7f350c0f1cf0 msgr2=0x7f350c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 shutdown_connections 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:15.620+0000 7f3510a14700 1 -- 172.21.15.114:0/1274310850 wait complete. 2024-02-20T13:29:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:16.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:15.021985+0000 mgr.smithi038.powepi (mgr.14156) 370 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:16.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:16 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:15.620402+0000 mon.smithi038 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.114:0/1274310850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:17.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:17.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:18.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:18 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:17.022396+0000 mgr.smithi038.powepi (mgr.14156) 371 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:19.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 -- 172.21.15.114:0/3243069935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 msgr2=0x7f21280f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:19.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 --2- 172.21.15.114:0/3243069935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f21280f56f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f211c009fb0 tx=0x7f211c009b30).stop 2024-02-20T13:29:19.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 -- 172.21.15.114:0/3243069935 shutdown_connections 2024-02-20T13:29:19.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 --2- 172.21.15.114:0/3243069935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f21280f56f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 -- 172.21.15.114:0/3243069935 >> 172.21.15.114:0/3243069935 conn(0x7f21280f0a30 msgr2=0x7f21280f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 -- 172.21.15.114:0/3243069935 shutdown_connections 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 -- 172.21.15.114:0/3243069935 wait complete. 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 Processor -- start 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.274+0000 7f212df01700 1 -- start start 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.275+0000 7f212df01700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f2128105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.275+0000 7f212df01700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2128105a00 con 0x7f21280f52d0 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.275+0000 7f212ceff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f2128105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.275+0000 7f212ceff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f2128105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:33036/0 (socket says 172.21.15.114:33036) 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.275+0000 7f212ceff700 1 -- 172.21.15.114:0/245412665 learned_addr learned my addr 172.21.15.114:0/245412665 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.275+0000 7f212ceff700 1 -- 172.21.15.114:0/245412665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c005040 con 0x7f21280f52d0 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.276+0000 7f212ceff700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f2128105510 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f211c000c00 tx=0x7f211c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.276+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211c0095f0 con 0x7f21280f52d0 2024-02-20T13:29:19.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.276+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2128105c00 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.276+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f211c007e50 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.276+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211c014e70 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.276+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2128106090 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.277+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f211c0079c0 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.277+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2128045bb0 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.277+0000 7f21257fa700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2118031090 0x7f2118033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.277+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f211c041c50 con 0x7f21280f52d0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.277+0000 7f2127fff700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2118031090 0x7f2118033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.277+0000 7f2127fff700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2118031090 0x7f2118033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:19.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.282+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f211c01b910 con 0x7f21280f52d0 2024-02-20T13:29:19.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.485+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2128106e90 con 0x7f21280f52d0 2024-02-20T13:29:19.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.486+0000 7f21257fa700 1 -- 172.21.15.114:0/245412665 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f211c01b2c0 con 0x7f21280f52d0 2024-02-20T13:29:19.488 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:19.488 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:19.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.487+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2118031090 msgr2=0x7f2118033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:19.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.487+0000 7f212df01700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2118031090 0x7f2118033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).stop 2024-02-20T13:29:19.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.487+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 msgr2=0x7f2128105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:19.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.487+0000 7f212df01700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f2128105510 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f211c000c00 tx=0x7f211c00b040).stop 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.488+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 shutdown_connections 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.488+0000 7f212df01700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f2118031090 0x7f2118033550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.488+0000 7f212df01700 1 --2- 172.21.15.114:0/245412665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21280f52d0 0x7f2128105510 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.488+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 >> 172.21.15.114:0/245412665 conn(0x7f21280f0a30 msgr2=0x7f21280f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.488+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 shutdown_connections 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:19.488+0000 7f212df01700 1 -- 172.21.15.114:0/245412665 wait complete. 2024-02-20T13:29:19.490 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:20.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:19.022802+0000 mgr.smithi038.powepi (mgr.14156) 372 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:20.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:20 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:19.487821+0000 mon.smithi038 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.114:0/245412665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:21.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:21.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:22.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:22 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:21.023143+0000 mgr.smithi038.powepi (mgr.14156) 373 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 -- 172.21.15.114:0/3427876434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 msgr2=0x7f73a00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 --2- 172.21.15.114:0/3427876434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a00f58a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7390009fb0 tx=0x7f7390009b30).stop 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 -- 172.21.15.114:0/3427876434 shutdown_connections 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 --2- 172.21.15.114:0/3427876434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a00f58a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 -- 172.21.15.114:0/3427876434 >> 172.21.15.114:0/3427876434 conn(0x7f73a00f0be0 msgr2=0x7f73a00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 -- 172.21.15.114:0/3427876434 shutdown_connections 2024-02-20T13:29:23.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.233+0000 7f73a82ab700 1 -- 172.21.15.114:0/3427876434 wait complete. 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a82ab700 1 Processor -- start 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a82ab700 1 -- start start 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a82ab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a82ab700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0060300 con 0x7f73a00f5480 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a6047700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a6047700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:33054/0 (socket says 172.21.15.114:33054) 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.234+0000 7f73a6047700 1 -- 172.21.15.114:0/98082049 learned_addr learned my addr 172.21.15.114:0/98082049 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f73a6047700 1 -- 172.21.15.114:0/98082049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390005040 con 0x7f73a00f5480 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f73a6047700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a0109510 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f73900089f0 tx=0x7f739000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7390009320 con 0x7f73a00f5480 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7390013ec0 con 0x7f73a00f5480 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a0060500 con 0x7f73a00f5480 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7390014d30 con 0x7f73a00f5480 2024-02-20T13:29:23.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.235+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a0109bc0 con 0x7f73a00f5480 2024-02-20T13:29:23.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.236+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f73900263a0 con 0x7f73a00f5480 2024-02-20T13:29:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.236+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a00591e0 con 0x7f73a00f5480 2024-02-20T13:29:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.236+0000 7f7396ffd700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f738c031090 0x7f738c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.236+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f73900423b0 con 0x7f73a00f5480 2024-02-20T13:29:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.236+0000 7f73a5846700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f738c031090 0x7f738c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.237+0000 7f73a5846700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f738c031090 0x7f738c033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f739c0096a0 tx=0x7f739c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:23.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.241+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f739001cb10 con 0x7f73a00f5480 2024-02-20T13:29:23.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.447+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73a0045bb0 con 0x7f73a00f5480 2024-02-20T13:29:23.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.447+0000 7f7396ffd700 1 -- 172.21.15.114:0/98082049 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f739001cb10 con 0x7f73a00f5480 2024-02-20T13:29:23.450 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:23.450 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:23.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f738c031090 msgr2=0x7f738c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:23.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f738c031090 0x7f738c033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f739c0096a0 tx=0x7f739c006b40).stop 2024-02-20T13:29:23.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 msgr2=0x7f73a0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:23.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a0109510 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f73900089f0 tx=0x7f739000b040).stop 2024-02-20T13:29:23.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 shutdown_connections 2024-02-20T13:29:23.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f738c031090 0x7f738c033550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:23.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 --2- 172.21.15.114:0/98082049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a00f5480 0x7f73a0109510 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:23.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 >> 172.21.15.114:0/98082049 conn(0x7f73a00f0be0 msgr2=0x7f73a00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:23.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 shutdown_connections 2024-02-20T13:29:23.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:23.449+0000 7f73a82ab700 1 -- 172.21.15.114:0/98082049 wait complete. 2024-02-20T13:29:23.453 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:24.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:23.023439+0000 mgr.smithi038.powepi (mgr.14156) 374 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:24.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:24 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:23.448969+0000 mon.smithi038 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.114:0/98082049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:25.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:25.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:26.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:26 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:25.023780+0000 mgr.smithi038.powepi (mgr.14156) 375 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 -- 172.21.15.114:0/1170060377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc0f6590 msgr2=0x7f3dcc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 --2- 172.21.15.114:0/1170060377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc0f6590 0x7f3dcc0f69b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3dc0004760 tx=0x7f3dc0009ad0).stop 2024-02-20T13:29:27.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 -- 172.21.15.114:0/1170060377 shutdown_connections 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 --2- 172.21.15.114:0/1170060377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc0f6590 0x7f3dcc0f69b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 -- 172.21.15.114:0/1170060377 >> 172.21.15.114:0/1170060377 conn(0x7f3dcc0f1cd0 msgr2=0x7f3dcc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 -- 172.21.15.114:0/1170060377 shutdown_connections 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.098+0000 7f3dd307a700 1 -- 172.21.15.114:0/1170060377 wait complete. 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd307a700 1 Processor -- start 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd307a700 1 -- start start 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd307a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 0x7f3dcc106d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd307a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc107200 con 0x7f3dcc1068f0 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd0e16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 0x7f3dcc106d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd0e16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 0x7f3dcc106d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58804/0 (socket says 172.21.15.114:58804) 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd0e16700 1 -- 172.21.15.114:0/4286151965 learned_addr learned my addr 172.21.15.114:0/4286151965 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.099+0000 7f3dd0e16700 1 -- 172.21.15.114:0/4286151965 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0009d70 con 0x7f3dcc1068f0 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.100+0000 7f3dd0e16700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 0x7f3dcc106d10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3dc0009820 tx=0x7f3dc0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.100+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dc000e040 con 0x7f3dcc1068f0 2024-02-20T13:29:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.100+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3dc00095b0 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.100+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dcc107400 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.100+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dc0014be0 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.100+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dcc107f50 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.101+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3dc0026070 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.101+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dcc0591e0 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.101+0000 7f3dc97fa700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3dbc031090 0x7f3dbc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.101+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3dc0041b60 con 0x7f3dcc1068f0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.101+0000 7f3dcbfff700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3dbc031090 0x7f3dbc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.102+0000 7f3dcbfff700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3dbc031090 0x7f3dbc033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3db80096a0 tx=0x7f3db8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:27.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.106+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3dc001e7e0 con 0x7f3dcc1068f0 2024-02-20T13:29:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.312+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3dcc045bb0 con 0x7f3dcc1068f0 2024-02-20T13:29:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.313+0000 7f3dc97fa700 1 -- 172.21.15.114:0/4286151965 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3dc001e7e0 con 0x7f3dcc1068f0 2024-02-20T13:29:27.315 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:27.315 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3dbc031090 msgr2=0x7f3dbc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3dbc031090 0x7f3dbc033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3db80096a0 tx=0x7f3db8006b40).stop 2024-02-20T13:29:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 msgr2=0x7f3dcc106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 0x7f3dcc106d10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3dc0009820 tx=0x7f3dc0009ad0).stop 2024-02-20T13:29:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 shutdown_connections 2024-02-20T13:29:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3dbc031090 0x7f3dbc033550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 --2- 172.21.15.114:0/4286151965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3dcc1068f0 0x7f3dcc106d10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 >> 172.21.15.114:0/4286151965 conn(0x7f3dcc0f1cd0 msgr2=0x7f3dcc0f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 shutdown_connections 2024-02-20T13:29:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:27.315+0000 7f3dd307a700 1 -- 172.21.15.114:0/4286151965 wait complete. 2024-02-20T13:29:27.318 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:27.024193+0000 mgr.smithi038.powepi (mgr.14156) 376 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:28.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:28 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:27.314848+0000 mon.smithi038 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.114:0/4286151965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:29.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:29.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:29.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:28.287920+0000 mon.smithi038 (mon.0) 333 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/trash_purge_schedule"}]: dispatch 2024-02-20T13:29:29.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:29 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:28.288151+0000 mon.smithi038 (mon.0) 334 : audit [INF] from='mgr.14156 172.21.15.38:0/1417782816' entity='mgr.smithi038.powepi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.powepi/mirror_snapshot_schedule"}]: dispatch 2024-02-20T13:29:30.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:30 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:29.024522+0000 mgr.smithi038.powepi (mgr.14156) 377 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:31.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 -- 172.21.15.114:0/2190148987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 msgr2=0x7f90540f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 --2- 172.21.15.114:0/2190148987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f90540f69b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f9048004760 tx=0x7f9048009ad0).stop 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 -- 172.21.15.114:0/2190148987 shutdown_connections 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 --2- 172.21.15.114:0/2190148987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f90540f69b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 -- 172.21.15.114:0/2190148987 >> 172.21.15.114:0/2190148987 conn(0x7f90540f1cd0 msgr2=0x7f90540f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 -- 172.21.15.114:0/2190148987 shutdown_connections 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.159+0000 7f905a1fb700 1 -- 172.21.15.114:0/2190148987 wait complete. 2024-02-20T13:29:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f905a1fb700 1 Processor -- start 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f905a1fb700 1 -- start start 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f905a1fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f905406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f905a1fb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9054069f00 con 0x7f90540f6590 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f90537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f905406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f90537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f905406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58824/0 (socket says 172.21.15.114:58824) 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.160+0000 7f90537fe700 1 -- 172.21.15.114:0/3052019697 learned_addr learned my addr 172.21.15.114:0/3052019697 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f90537fe700 1 -- 172.21.15.114:0/3052019697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048009d70 con 0x7f90540f6590 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f90537fe700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f905406ba80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f9048004170 tx=0x7f9048009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f904800e040 con 0x7f90540f6590 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905406a0a0 con 0x7f90540f6590 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9048008ef0 con 0x7f90540f6590 2024-02-20T13:29:31.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9048014c90 con 0x7f90540f6590 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.161+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905406a530 con 0x7f90540f6590 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.162+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9048014420 con 0x7f90540f6590 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.162+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90540591e0 con 0x7f90540f6590 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.162+0000 7f903bfff700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f903c031090 0x7f903c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.162+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9048042300 con 0x7f90540f6590 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.162+0000 7f9052ffd700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f903c031090 0x7f903c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:31.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.163+0000 7f9052ffd700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f903c031090 0x7f903c033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f90440096a0 tx=0x7f9044006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:31.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.167+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9048026300 con 0x7f90540f6590 2024-02-20T13:29:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.376+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90540fc080 con 0x7f90540f6590 2024-02-20T13:29:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.377+0000 7f903bfff700 1 -- 172.21.15.114:0/3052019697 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9048012b20 con 0x7f90540f6590 2024-02-20T13:29:31.379 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:31.379 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f903c031090 msgr2=0x7f903c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f903c031090 0x7f903c033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f90440096a0 tx=0x7f9044006b40).stop 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 msgr2=0x7f905406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f905406ba80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f9048004170 tx=0x7f9048009ad0).stop 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 shutdown_connections 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f903c031090 0x7f903c033550 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 --2- 172.21.15.114:0/3052019697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90540f6590 0x7f905406ba80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 >> 172.21.15.114:0/3052019697 conn(0x7f90540f1cd0 msgr2=0x7f90540f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 shutdown_connections 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:31.379+0000 7f905a1fb700 1 -- 172.21.15.114:0/3052019697 wait complete. 2024-02-20T13:29:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:32.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:31.024850+0000 mgr.smithi038.powepi (mgr.14156) 378 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:32.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:32 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:31.378818+0000 mon.smithi038 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.114:0/3052019697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:33.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T13:29:33.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 24090638-cff2-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T13:29:34.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:34 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:33.025221+0000 mgr.smithi038.powepi (mgr.14156) 379 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 -- 172.21.15.114:0/1028016581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 msgr2=0x7f3d680f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 --2- 172.21.15.114:0/1028016581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d680f4c70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f3d58009fb0 tx=0x7f3d58009b30).stop 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 -- 172.21.15.114:0/1028016581 shutdown_connections 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 --2- 172.21.15.114:0/1028016581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d680f4c70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 -- 172.21.15.114:0/1028016581 >> 172.21.15.114:0/1028016581 conn(0x7f3d680f0390 msgr2=0x7f3d680f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 -- 172.21.15.114:0/1028016581 shutdown_connections 2024-02-20T13:29:35.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.019+0000 7f3d6d68c700 1 -- 172.21.15.114:0/1028016581 wait complete. 2024-02-20T13:29:35.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d6d68c700 1 Processor -- start 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d6d68c700 1 -- start start 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d6d68c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d681067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d6d68c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68106c90 con 0x7f3d680f4850 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d66ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d681067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d66ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d681067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.114:58836/0 (socket says 172.21.15.114:58836) 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d66ffd700 1 -- 172.21.15.114:0/804682624 learned_addr learned my addr 172.21.15.114:0/804682624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d66ffd700 1 -- 172.21.15.114:0/804682624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d58005040 con 0x7f3d680f4850 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.020+0000 7f3d66ffd700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d681067a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f3d58000c00 tx=0x7f3d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.021+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d580095f0 con 0x7f3d680f4850 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.021+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d68106e90 con 0x7f3d680f4850 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.021+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3d58007e50 con 0x7f3d680f4850 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.021+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d58014e70 con 0x7f3d680f4850 2024-02-20T13:29:35.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.021+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d68107320 con 0x7f3d680f4850 2024-02-20T13:29:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.021+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3d58025070 con 0x7f3d680f4850 2024-02-20T13:29:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.022+0000 7f3d4f7fe700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d50031090 0x7f3d50033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T13:29:35.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.022+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3d58041e30 con 0x7f3d680f4850 2024-02-20T13:29:35.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.022+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68045bb0 con 0x7f3d680f4850 2024-02-20T13:29:35.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.022+0000 7f3d667fc700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d50031090 0x7f3d50033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T13:29:35.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.022+0000 7f3d667fc700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d50031090 0x7f3d50033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f3d5c0096a0 tx=0x7f3d5c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T13:29:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.028+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3d5800f4e0 con 0x7f3d680f4850 2024-02-20T13:29:35.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.232+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d68107fd0 con 0x7f3d680f4850 2024-02-20T13:29:35.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.233+0000 7f3d4f7fe700 1 -- 172.21.15.114:0/804682624 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d5800f300 con 0x7f3d680f4850 2024-02-20T13:29:35.235 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-20T13:29:35.235 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"24090638-cff2-11ee-95bb-87774f69a715","modified":"2024-02-20T13:16:19.788530Z","created":"2024-02-20T13:16:19.788530Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d50031090 msgr2=0x7f3d50033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d50031090 0x7f3d50033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f3d5c0096a0 tx=0x7f3d5c006b40).stop 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 msgr2=0x7f3d681067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d681067a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f3d58000c00 tx=0x7f3d5800b040).stop 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 shutdown_connections 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:6800/175894382,v1:172.21.15.38:6801/175894382] conn(0x7f3d50031090 0x7f3d50033550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 --2- 172.21.15.114:0/804682624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d680f4850 0x7f3d681067a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 >> 172.21.15.114:0/804682624 conn(0x7f3d680f0390 msgr2=0x7f3d680f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 shutdown_connections 2024-02-20T13:29:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-20T13:29:35.234+0000 7f3d6d68c700 1 -- 172.21.15.114:0/804682624 wait complete. 2024-02-20T13:29:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-02-20T13:29:35.695 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:35 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: audit 2024-02-20T13:29:35.234326+0000 mon.smithi038 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.114:0/804682624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T13:29:36.027 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-20T13:29:36.029 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-20T13:29:36.029 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T13:29:36.056 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T13:29:36.077 INFO:tasks.cephadm:Stopping all daemons... 2024-02-20T13:29:36.078 INFO:tasks.cephadm.mon.smithi038:Stopping mon.smithi038... 2024-02-20T13:29:36.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038 2024-02-20T13:29:36.402 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:36 smithi038 systemd[1]: Stopping Ceph mon.smithi038 for 24090638-cff2-11ee-95bb-87774f69a715... 2024-02-20T13:29:36.402 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:36 smithi038 podman[44880]: Error: no container with name or ID "ceph-24090638-cff2-11ee-95bb-87774f69a715-mon.smithi038" found: no such container 2024-02-20T13:29:36.402 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: cluster 2024-02-20T13:29:35.025488+0000 mgr.smithi038.powepi (mgr.14156) 380 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T13:29:36.402 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:29:36.401+0000 7f2bc73a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi038 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T13:29:36.696 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:36 smithi038 ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038[38379]: debug 2024-02-20T13:29:36.401+0000 7f2bc73a6700 -1 mon.smithi038@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T13:29:37.196 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Feb 20 13:29:37 smithi038 bash[44921]: ceph-24090638-cff2-11ee-95bb-87774f69a715-mon-smithi038 2024-02-20T13:29:37.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi038.service' 2024-02-20T13:29:37.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-20T13:29:37.397 INFO:tasks.cephadm.mon.smithi038:Stopped mon.smithi038 2024-02-20T13:29:37.397 INFO:tasks.cephadm.mon.smithi114:Stopping mon.smithi114... 2024-02-20T13:29:37.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114 2024-02-20T13:29:37.434 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114.service: Unit ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114.service not loaded. 2024-02-20T13:29:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-20T13:29:37.436 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi114" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 5: 'sudo systemctl stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114' 2024-02-20T13:29:37.438 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-20T13:29:37.439 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/remote/smithi038/crash 2024-02-20T13:29:37.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/crash -- . 2024-02-20T13:29:37.489 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/remote/smithi114/crash 2024-02-20T13:29:37.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/crash -- . 2024-02-20T13:29:37.535 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/24090638-cff2-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-20T13:29:37.535 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-02-20T13:29:37.536 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-20T13:29:37.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-20T13:29:37.570 INFO:teuthology.orchestra.run.smithi038.stderr:grep: /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph.log: No such file or directory 2024-02-20T13:29:37.572 INFO:tasks.cephadm:Compressing logs... 2024-02-20T13:29:37.573 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T13:29:37.615 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T13:29:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T13:29:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:: No such file or directory 2024-02-20T13:29:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T13:29:37.644 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-mon.smithi038.log 2024-02-20T13:29:37.644 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-mgr.smithi038.powepi.log 2024-02-20T13:29:37.644 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-volume.log: /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-mon.smithi038.log: /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-mgr.smithi038.powepi.log: 90.2% -- replaced with /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T13:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-20T13:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T13:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T13:29:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T13:29:37.650 INFO:teuthology.orchestra.run.smithi114.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T13:29:37.650 INFO:teuthology.orchestra.run.smithi038.stderr: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T13:29:37.651 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-02-20T13:29:37.652 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m0.018s 2024-02-20T13:29:37.652 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m0.013s 2024-02-20T13:29:37.652 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.015s 2024-02-20T13:29:37.742 INFO:teuthology.orchestra.run.smithi038.stderr: 92.7% -- replaced with /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-mgr.smithi038.powepi.log.gz 2024-02-20T13:29:37.823 INFO:teuthology.orchestra.run.smithi038.stderr: 90.6% -- replaced with /var/log/ceph/24090638-cff2-11ee-95bb-87774f69a715/ceph-mon.smithi038.log.gz 2024-02-20T13:29:37.824 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-02-20T13:29:37.824 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m0.193s 2024-02-20T13:29:37.825 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m0.282s 2024-02-20T13:29:37.825 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.026s 2024-02-20T13:29:37.825 INFO:tasks.cephadm:Archiving logs... 2024-02-20T13:29:37.825 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/remote/smithi038/log 2024-02-20T13:29:37.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T13:29:37.951 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/remote/smithi114/log 2024-02-20T13:29:37.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T13:29:37.986 INFO:tasks.cephadm:Removing cluster... 2024-02-20T13:29:37.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 24090638-cff2-11ee-95bb-87774f69a715 --force 2024-02-20T13:32:44.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 24090638-cff2-11ee-95bb-87774f69a715 --force 2024-02-20T13:32:44.754 INFO:tasks.cephadm:Removing cephadm ... 2024-02-20T13:32:44.754 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T13:32:44.774 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T13:32:44.792 INFO:tasks.cephadm:Teardown complete 2024-02-20T13:32:44.793 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 5: 'sudo systemctl stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114' 2024-02-20T13:32:45.017 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e1690a0211724cf487a7e16e944398e9 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b0bd01b8442d66b8a31f8cb3050334b026de2f3b/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 5: 'sudo systemctl stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114' 2024-02-20T13:32:45.022 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-20T13:32:45.031 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-20T13:32:45.039 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-20T13:32:45.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T13:32:45.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T13:32:45.139 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-20T13:32:45.149 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-20T13:32:45.149 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:32:45.152 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T13:32:45.170 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2024-02-20T13:32:45.170 INFO:teuthology.orchestra.run.smithi114.stderr:bash: ntpq: command not found 2024-02-20T13:32:45.173 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:32:45.174 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-02-20T13:32:45.174 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +279us[ +279us] +/- 83ms 2024-02-20T13:32:45.174 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -600us[ -624us] +/- 60ms 2024-02-20T13:32:45.174 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 93 +524us[ +502us] +/- 84ms 2024-02-20T13:32:45.174 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:32:45.176 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T13:32:45.176 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-02-20T13:32:45.176 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +270us[ +270us] +/- 82ms 2024-02-20T13:32:45.176 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -592us[ -582us] +/- 60ms 2024-02-20T13:32:45.176 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +541us[ +552us] +/- 85ms 2024-02-20T13:32:45.176 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T13:32:45.177 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-20T13:32:45.186 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-20T13:32:45.187 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-20T13:32:45.197 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T13:32:45.283 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T13:32:45.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:32:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:32:45.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T13:32:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:32:45.410 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-20T13:32:45.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-20T13:32:45.429 INFO:teuthology.task.internal:Duration was 1652.282240 seconds 2024-02-20T13:32:45.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-20T13:32:45.439 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-20T13:32:45.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T13:32:45.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T13:32:45.490 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:32:45.496 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T13:32:45.881 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-20T13:32:45.882 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-02-20T13:32:45.882 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T13:32:45.908 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-02-20T13:32:45.908 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T13:32:45.930 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-20T13:32:45.931 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T13:32:45.950 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T13:32:46.022 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-20T13:32:46.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T13:32:46.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T13:32:46.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-20T13:32:46.146 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-20T13:32:46.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T13:32:46.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T13:32:46.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-20T13:32:46.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:32:46.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:32:46.242 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-02-20T13:32:46.267 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-02-20T13:32:46.299 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:32:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:32:46.342 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T13:32:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T13:32:46.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-20T13:32:46.369 INFO:teuthology.task.internal:Transferring archived files... 2024-02-20T13:32:46.371 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/remote/smithi038 2024-02-20T13:32:46.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T13:32:46.424 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709/remote/smithi114 2024-02-20T13:32:46.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T13:32:46.464 INFO:teuthology.task.internal:Removing archive directory... 2024-02-20T13:32:46.465 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T13:32:46.467 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T13:32:46.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-20T13:32:46.528 INFO:teuthology.task.internal:Not uploading archives. 2024-02-20T13:32:46.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-20T13:32:46.584 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-20T13:32:46.584 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T13:32:46.586 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T13:32:46.602 INFO:teuthology.orchestra.run.smithi038.stdout: 526077 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 13:32 /home/ubuntu/cephtest 2024-02-20T13:32:46.604 INFO:teuthology.orchestra.run.smithi114.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 13:32 /home/ubuntu/cephtest 2024-02-20T13:32:46.605 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-20T13:32:46.615 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-20T13:32:46.635 INFO:teuthology.nuke:Checking targets against current locks 2024-02-20T13:32:46.662 DEBUG:teuthology.nuke:shortname: smithi038 2024-02-20T13:32:46.663 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T13:32:46.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 12:57:04.711169', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQLIMGqyei2iIGo1IJY4+EnIdvWofaXj/gZE0rzlnawmU5fH6mPaRsy3FVFAsVUI5nf5KM68XoxSZ0jdlO9EHs='} 2024-02-20T13:32:46.689 DEBUG:teuthology.nuke:shortname: smithi114 2024-02-20T13:32:46.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T13:32:46.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_20:21:18-fs-wip-yuri5-testing-2024-02-16-1120-quincy-distro-default-smithi/7567709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 12:57:04.713052', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOl5YEEZOed9gZqRtxggC/V12jeAgn4UTiu1Ju7iQCdDRqwJjcyRFk7QyIF2U2IKM8A1RS5s7kPBZHIUxLcHLc='} 2024-02-20T13:32:46.731 INFO:teuthology.orchestra.console.smithi038:Power off 2024-02-20T13:32:46.731 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T13:32:46.757 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-02-20T13:32:46.757 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:46.776 INFO:teuthology.orchestra.console.smithi114:Power off 2024-02-20T13:32:46.776 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T13:32:46.786 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-02-20T13:32:46.800 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-02-20T13:32:46.800 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:46.826 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-02-20T13:32:50.787 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:50.827 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:50.898 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-02-20T13:32:50.938 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-02-20T13:32:54.900 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:54.939 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:55.011 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-02-20T13:32:55.050 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-02-20T13:32:55.151 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-02-20T13:32:59.016 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T13:32:59.127 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-02-20T13:32:59.228 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-02-20T13:32:59.334 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1652.2822403907776 failure_reason: 'Command failed on smithi114 with status 5: ''sudo systemctl stop ceph-24090638-cff2-11ee-95bb-87774f69a715@mon.smithi114''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e1690a0211724cf487a7e16e944398e9 status: fail success: false 2024-02-20T13:32:59.334 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T13:32:59.424 INFO:teuthology.run:FAIL